Omnidirectional camera system with improved point of interest selection

ABSTRACT

An image capturing device includes an imaging device and circuitry. The imaging device captures an image. The circuitry defines a point of interest in the image, converts the defined point of interest in accordance with attitude information of the image capturing device, and cuts out a viewable area from the image. The viewable area includes the converted point of interest.

CROSS-REFERENCE TO RELATED APPLICATION

This patent application is based on and claims priority pursuant to 35U.S.C. § 119(a) to Japanese Patent Application No. 2019-049128 filed onMar. 15, 2019 in the Japan Patent Office, the entire disclosure of whichis hereby incorporated by reference herein.

BACKGROUND Technical Field

The present invention relates to an image capturing device, an imageprocessing system, and an image processing method.

Description of the Related Art

There is an image capturing device that captures a 360-degree,all-directional (hereinafter referred to as omnidirectional) image ofsurroundings at one time with a plurality of wide-angle lenses such asfisheye lenses or ultrawide-angle lenses. The image capturing devicegenerates the omnidirectional image by projecting images from therespective lenses onto imaging elements and combining the projectedimages through image processing. For example, the image capturing devicegenerates the omnidirectional image by capturing the image of anomnidirectional subject with two wide-angle lenses each having an angleof view exceeding 180 degrees.

Whereas the omnidirectional image generated by the image capturingdevice is expressed in a spherical coordinate system, a typical displayhas a planar surface and thus has difficulty in displaying the entireomnidirectional image at one time. To display the omnidirectional image,therefore, a dedicated viewer application is normally used to convert apart of the omnidirectional image with a certain angle of view into animage suitable for the typical display.

There is a technique of determining a point of interest to enable aterminal to display a part of the omnidirectional image with a givenangle of view. According to the technique, in response to tilting orrotation of the image capturing device by a user, the point of interestis changed, and the changed point of interest is displayed on thetypical display.

SUMMARY

In one embodiment of this invention, there is provided an improved imagecapturing device that includes, for example, an imaging device andcircuitry. The imaging device captures an image. The circuitry defines apoint of interest in the image, converts the defined point of interestin accordance with attitude information of the image capturing device,and cuts out a viewable area from the image. The viewable area includesthe converted point of interest.

In one embodiment of this invention, there is provided an improved imageprocessing system that includes, for example, circuitry. The circuitryacquires an image captured by an image capturing device, defines a pointof interest in the image, converts the defined point of interest inaccordance with attitude information of the image capturing device, cutsout, from the image, a viewable area including the converted point ofinterest, and displays, in a display area of a display, the viewablearea cut out from the image and including the converted point ofinterest.

In one embodiment of this invention, there is provided an improved imageprocessing method that includes, for example, acquiring an imagecaptured by an image capturing device, acquiring attitude information ofthe image capturing device, converting a defined point of interest inaccordance with the attitude information, cutting out, from the image, aviewable area including the converted point of interest, and displaying,in a display area of a display, the viewable area cut out from the imageand including the converted point of interest.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

A more complete appreciation of the disclosure and many of the attendantadvantages and features thereof can be readily obtained and understoodfrom the following detailed description with reference to theaccompanying drawings, wherein:

FIG. 1 is a diagram illustrating a schematic operation of an imagecommunication system of an embodiment of the present invention totransmit from one site to another site an omnidirectional image with apoint of interest specified in a given direction;

FIG. 2A is a right side view of an image capturing device included inthe image communication system;

FIG. 2B is a front view of the image capturing device;

FIG. 2C is a plan view of the image capturing device;

FIG. 3 is a conceptual diagram illustrating use of the image capturingdevice;

FIG. 4A is a diagram illustrating a front hemispherical image capturedby the image capturing device;

FIG. 4B is a diagram illustrating a rear hemispherical image captured bythe image capturing device;

FIG. 4C is a diagram illustrating an equidistant cylindrical imagegenerated from the hemispherical images by equidistant cylindricalprojection;

FIG. 5A is a conceptual diagram illustrating the equidistant cylindricalimage covering a sphere;

FIG. 5B is a diagram illustrating an omnidirectional image obtained fromthe equidistant cylindrical image;

FIG. 6 is a diagram illustrating respective positions of a virtualcamera and a predetermined area of the omnidirectional image when theomnidirectional image is expressed as a three-dimensional solid sphere;

FIG. 7A is a perspective view of the omnidirectional image in FIG. 6 asthe solid sphere;

FIG. 7B is a diagram illustrating an image of the predetermined areadisplayed on a display of a communication terminal included in the imagecommunication system;

FIG. 8 is a diagram illustrating the relationship between predeterminedarea information and the image of the predetermined area;

FIG. 9 is a diagram illustrating a point in a three-dimensionalEuclidean space represented by spherical coordinates;

FIG. 10 is a schematic diagram illustrating a configuration of the imagecommunication system of the embodiment;

FIG. 11 is a diagram illustrating an exemplary hardware configuration ofthe image capturing device;

FIG. 12 is a diagram illustrating an exemplary hardware configuration ofcommunication terminals included in the image communication system;

FIG. 13 is a diagram illustrating an exemplary hardware configuration ofa communication management system included in the image communicationsystem;

FIG. 14 is a diagram illustrating an exemplary hardware configuration ofother communication terminals included in the image communicationsystem;

FIGS. 15A, 15B (FIG. 15), 16A, and 16B (FIG. 16) are functional blockdiagrams illustrating exemplary functional blocks of the imagecommunication system;

FIG. 17 is a conceptual diagram illustrating an image type managementtable stored in a communication terminal of the image communicationsystem;

FIG. 18 is a conceptual diagram illustrating an image capturing devicemanagement table stored in the communication terminal;

FIG. 19 is a conceptual diagram illustrating a predetermined areainformation management table stored in the communication terminal;

FIG. 20 is a conceptual diagram illustrating a session management tablestored in the communication management system;

FIG. 21 is a conceptual diagram illustrating an image type managementtable stored in the communication management system;

FIG. 22 is a conceptual diagram illustrating a predetermined areainformation management table stored in the communication managementsystem;

FIG. 23 is a sequence diagram illustrating an example of a process ofhaving a communication terminal of the image communication systemparticipate in a specific communication session;

FIG. 24 is a diagram illustrating an example of a selection screendisplayed on a communication terminal of the image communication systemto select a communication session in a virtual meeting room;

FIG. 25 is a diagram illustrating an example of coordinate axes of theimage capturing device;

FIG. 26 is a diagram illustrating an example of a reference attitude ofthe image capturing device;

FIGS. 27A to 27C are diagrams illustrating an example of values detectedby an acceleration and orientation sensor of the image capturing device;

FIG. 28A is a conceptual diagram illustrating a state in which aphotographer points the image capturing device at an object;

FIG. 28B is a diagram illustrating an example of an equidistantcylindrical image as an omnidirectional image expressed by equidistantcylindrical projection without zenith correction;

FIG. 28C is a diagram illustrating an example of an equidistantcylindrical image as an omnidirectional image expressed by equidistantcylindrical projection with zenith correction;

FIG. 29 is a functional block diagram illustrating exemplary functionalblocks of an image processing unit of the image capturing device;

FIG. 30A is a diagram illustrating an example of a conversion table usedby the image capturing device;

FIG. 30B is a diagram illustrating an example of conversion from a planecoordinate system into a spherical coordinate system;

FIG. 31 is a diagram illustrating an example of correction of theconversion table based on attitude information of the image capturingdevice;

FIG. 32 is a diagram schematically illustrating an example of an imagerotation process executed by an image rotating unit of the imagecapturing device;

FIG. 33 is a sequence diagram illustrating an exemplary process ofcommunicating an omnidirectional image and audio data in a video call inthe image communication system;

FIG. 34A is a flowchart illustrating an exemplary process of generatingthe omnidirectional image performed by the image capturing device;

FIG. 34B is a flowchart illustrating an exemplary process of displayingthe omnidirectional image performed by the communication terminals ofthe image communication system; and

FIG. 35 is a diagram illustrating an example of an image display screendisplayed on a display of a communication terminal of the imagecommunication system.

The accompanying drawings are intended to depict embodiments of thepresent invention and should not be interpreted to limit the scopethereof. The accompanying drawings are not to be considered as drawn toscale unless explicitly noted.

DETAILED DESCRIPTION

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the presentinvention. As used herein, the singular forms “a”, “an” and “the” areintended to include the plural forms as well, unless the context clearlyindicates otherwise. In the drawings illustrating embodiments of thepresent invention, members or components having the same function orshape will be denoted with the same reference numerals to avoidredundant description.

In describing embodiments illustrated in the drawings, specificterminology is employed for the sake of clarity. However, the disclosureof this specification is not intended to be limited to the specificterminology so selected and it is to be understood that each specificelement includes all technical equivalents that have a similar function,operate in a similar manner, and achieve a similar result.

As described above, the existing technique changes the point of interestin response to tilting or rotation of the image capturing device by auser. According to the technique, however, it is difficult for the userof the image capturing device (i.e., a photographer) to specify thepoint of interest in real time. The photographer may want to specify thepoint of interest in real time in some situations such as in viewing ofreal estate properties, for example.

When applied to viewing of real estate properties, the image capturingdevice is capable of capturing the image of the entire interior of aproperty, enabling an interested client to grasp a general idea of theproperty without physically visiting the property and thus save time andtrouble for transportation.

For example, viewing of a real estate property may proceed as follows. Areal estate agent visits the property and captures the image of theproperty with the image capturing device. Using a communicationterminal, an interested client at a remote site receives and views, inreal time, the image of the property captured by the image capturingdevice. As well as the image, sound is transmitted to the communicationterminal of the client, enabling the client to view the property whileconversing with the agent.

The image transmitted to the communication terminal of the client,however, is limited to a part of the omnidirectional image with acertain angle of view. Therefore, the client is not necessarily watchinga part of the omnidirectional image being explained by the agent.Further, it is difficult to immediately display the part on aflat-surface device. Consequently, the client is confused about whichpart of the omnidirectional image is to be looked at. For instance, whenthe agent inside the property says “This room is equipped with thislatest model of air-conditioner,” the client viewing from the remotesite is not necessarily looking in the direction of the air-conditionerin the omnidirectional image, making it difficult for both the agent andthe client to well communicate on the point of interest specified inreal time.

To address this issue, the agent may display a preview of theomnidirectional image on a communication terminal of the agent andperform an operation of specifying the point of interest, to therebytransmit the data of the point of interest from the communicationterminal of the agent to the communication terminal of the clientviewing from the remote site such that the agent and the client sharethe same point of interest. However, it is difficult for the agent tooperate the communication terminal while capturing the image with theimage capturing device, particularly when the number of agents availableis limited to one.

There is also a technique of estimating the point of interest byanalyzing the omnidirectional image. The technique, however, is based onthe estimation of the point of interest, and thus the part to beexplained by the agent is not necessarily estimated as the point ofinterest.

The present invention provides an image processing system enabling thephotographer to specify the point of interest in real time.

An image processing system according to an embodiment of the presentinvention and an image processing method executed by the imageprocessing system will be described below with reference to thedrawings.

A schematic operation of an image communication system 10 (e.g., anexample of the image processing system) will first be described withFIG. 1.

FIG. 1 is a diagram illustrating a schematic operation of the imagecommunication system 10 to transmit from a site A to a site B anomnidirectional image with a point of interest specified in a givendirection.

The coordinates of the point of interest are previously defined in animage capturing device 5 a according to the embodiment. For example, thecoordinates of the point of interest are expressed as (x, y), whichrepresent pixels or an area of an imaging element of the image capturingdevice 5 a corresponding to the image of a subject located on the upperside in the longitudinal direction of the image capturing device 5 a.The coordinates of the point of interest are coordinates on the imagingelement, and are preset in a shipment process of the image capturingdevice 5 a.

When a photographer 8 at the site A wants a user 9 b at the site B topay attention to an air-conditioner 140, the photographer 8 points theupper side in the longitudinal direction of the image capturing device 5a at the air-conditioner 140. The image capturing device 5 a constantlyacquires attitude information for later-described zenith correction. Theimage capturing device 5 a captures the omnidirectional image andperforms the zenith correction based on the attitude information.Therefore, the image capturing device 5 a does not capture the image ofthe air-conditioner 140 at an angle. Further, the image of theair-conditioner 140 is on the coordinates of the point of interest,i.e., the air-conditioner 140 is a subject aligned with the coordinatesof the point of interest. The attitude information represents how thecoordinates of the point of interest are changed by tilting of the imagecapturing device 5 a by the photographer 8. With the zenith correction,therefore, the coordinates of the point of interest converted from thedefined coordinates of the point of interest are identified.

Via a communication terminal 1 and a communication network 100, theimage capturing device 5 a transmits the omnidirectional image and theconverted coordinates of the point of interest to a communicationterminal 2 at the site B.

The communication terminal 2 then generates a predetermined area imageby cutting out from the omnidirectional image a predetermined areaincluding the coordinates of the point of interest and displays thegenerated predetermined area image on a display, for example,irrespective of the predetermined area image displayed by thecommunication terminal 2 until the receipt of the omnidirectional imageand the coordinates of the point of interest. Thereby, the photographer8 is able to draw the attention of the user 9 b to the object to whichthe photographer 8 wants the user 9 b to pay attention. Further, aviewer (i.e., the user 9 b) is unable to change, for at least a certaintime, the predetermined area image generated based on point-of-interestinformation including the coordinates of the point of interest. That is,the predetermined area image generated based on the point-of-interestinformation is forcibly displayed.

As described above, according to the image communication system 10 ofthe embodiment, the coordinates of the point of interest are previouslydefined in the image capturing device 5 a. When the photographer 8points the image capturing device 5 a at the target object such that thecoordinates of the point of interest are aligned with the target object,the coordinates of the point of interest are converted in accordancewith the attitude information, and the converted coordinates of thepoint of interest are transmitted to the user 9 b. Consequently, thephotographer 8 is able to draw the attention of the user 9 b at the siteB to the point of interest in real time.

As described above, the defined point of interest refers to a point orarea on the imaging element. The subject photographed by the imagecapturing device 5 a is constantly located in the same direction asviewed from the image capturing device 5 a. In the present embodiment,the defined point of interest is set to pixels or an area on the imagingelement corresponding to the image of the subject located on the upperside in the longitudinal direction of the image capturing device 5 a.The point of interest, however, may be set to a desired area. Forexample, the point of interest may be set to pixels of the imagingelement corresponding to the image of the subject located in thedirection pointed by the lower side in the longitudinal direction of theimage capturing device 5 a, the direction of the optical axis of a lensof the image capturing device 5 a, or the direction pointed by aprojecting portion of the image capturing device 5 a.

Further, in the present embodiment, the predetermined area refers to anarea of the omnidirectional image viewable to a user. The predeterminedarea depends on the viewpoint of the user and the angle of view of animage capturing device. The predetermined area may be a previously setarea or an area to be selected for display by the user. Since the term“predetermined area” used here refers to the area viewable to a user,the predetermined area may also be described as the viewable area.

The attitude information of an image capturing device is any informationcapable of identifying a given direction pointed by the image capturingdevice. For example, the attitude information may be information of thedegree of tilt of the image capturing device from the image capturingdevice in the erected state thereof or information related to rotationof the image capturing device around the axes in a three-dimensionalspace.

A method of generating the omnidirectional image will be described withFIG. 2A to FIG. 9.

The exterior of the image capturing device 5 a will first be describedwith FIGS. 2A to 2C.

The image capturing device 5 a is a digital camera for capturing animage to generate a three-dimensional, 360-degree omnidirectional imagebased on the captured image. FIG. 2A is a right side view of the imagecapturing device 5 a. FIG. 2B is a front view of the image capturingdevice 5 a. FIG. 2C is a plan view of the image capturing device 5 a.

As illustrated in FIG. 2A, the image capturing device 5 a has a sizesuitable for being held by a human hand. Further, as illustrated inFIGS. 2A to 2C, an upper portion of the image capturing device 5 a isequipped with imaging elements 103 a and 103 b, which are formed on onesurface and the other surface, respectively, of the image capturingdevice 5 a. Each of the imaging elements 103 a and 103 b is implementedby an image sensor, and is used with an optical member (e.g., a fisheyelens 102 a or 102 b in FIG. 11) capable of capturing a hemisphericalimage with an angle of view of at least 180 degrees. Further, asillustrated in FIG. 2B, the surface of the image capturing device 5 awith the imaging element 103 b is equipped with an operation device 115including a shutter button SB (see FIGS. 25 and 26). As well as theshutter button SB, other buttons such as a wireless fidelity (Wi-Fi,registered trademark) button and a shooting mode switching button mayalso be formed on any surface of the image capturing device 5 a such asa side surface of the image capturing device 5 a.

With reference to FIG. 3, a description will be given of a situation inwhich the image capturing device 5 a is used.

FIG. 3 is a conceptual diagram illustrating use of the image capturingdevice 5 a. As illustrated in FIG. 3, the image capturing device 5 a isused as held by a hand of a user to capture the image of a subjectaround the user. In this case, the image of the subject around the useris captured by the imaging elements 103 a and 103 b illustrated in FIGS.2A to 2C to obtain two hemispherical images.

With reference to FIG. 4A to FIG. 5B, a description will be given of anoverview of a process of generating the omnidirectional image from theimages captured by the image capturing device 5 a.

FIG. 4A is a diagram illustrating a front hemispherical image capturedby the image capturing device 5 a. FIG. 4B is a diagram illustrating arear hemispherical image captured by the image capturing device 5 a.FIG. 4C is a diagram illustrating an image generated from thehemispherical images by equidistant cylindrical projection (hereinafterreferred to as the equidistant cylindrical image). FIG. 5A is aconceptual diagram illustrating the equidistant cylindrical imagecovering a sphere. FIG. 5B is a diagram illustrating an omnidirectionalimage obtained from the equidistant cylindrical image.

As illustrated in FIG. 4A, the front hemispherical image captured by theimaging element 103 a is distorted by the fisheye lens 102 a. Further,as illustrated in FIG. 4B, the rear hemispherical image captured by theimaging element 103 b is distorted by the fisheye lens 102 b. The imagecapturing device 5 a combines the front hemispherical image and the rearhemispherical image rotated therefrom by 180 degrees, to therebygenerate an equidistant cylindrical image, as illustrated in FIG. 4C.

Then, with an application programming interface (API) such as opengraphics library for embedded systems (OpenGL ES, registered trademark),the equidistant cylindrical image is placed to the surface of a sphereto cover the spherical surface, as illustrated in FIG. 5A. Thereby, theomnidirectional image CE as illustrated in FIG. 5B is generated. Theomnidirectional image CE is thus expressed as the equidistantcylindrical image facing the center of the sphere. OpenGL ES is agraphics library application used to visualize two-dimensional orthree-dimensional data. The omnidirectional image CE may be a still orvideo image.

As described above, the omnidirectional image CE is obtained as theimage placed on a sphere to cover the spherical surface, and thus isperceived as unnatural to human eyes. Therefore, the predetermined areaimage of the predetermined area as a part of the omnidirectional imageCE is displayed as a planar image with less distortion to be perceivedas less unnatural to human eyes.

Display of the predetermined area image will be described with FIG. 6 toFIG. 7B.

FIG. 6 is a diagram illustrating respective positions of a virtualcamera IC and a predetermined area T when the omnidirectional image CEis expressed as a three-dimensional solid sphere CS. The position of thevirtual camera IC corresponds to the position of the viewpoint of a userviewing the omnidirectional image CE expressed as the three-dimensionalsolid sphere CS. FIG. 7A is a perspective view of the omnidirectionalimage CE in FIG. 6 as the solid sphere CS. FIG. 7B is a diagramillustrating a predetermined area image Q displayed on a display. InFIG. 7A, the omnidirectional image CE in FIG. 6 is illustrated as thethree-dimensional solid sphere CS. When the omnidirectional image CEgenerated as described above is expressed as the solid sphere CS, thevirtual camera IC is located outside the omnidirectional image CE, asillustrated in FIG. 6. The predetermined area T of the omnidirectionalimage CE corresponds to an imaging area of the virtual camera IC, and isidentified by predetermined area information. The predetermined areainformation represents the imaging direction and the angle of view ofthe virtual camera IC in a three-dimensional virtual space including theomnidirectional image CE.

As illustrated in FIG. 7B, the predetermined area T in FIG. 7A isdisplayed on a predetermined display as the image of the imaging area ofthe virtual camera IC. The image illustrated in FIG. 7B is thepredetermined area image Q expressed by initially set predetermined areainformation, for example. The predetermined area information may beexpressed not as the position coordinates of the virtual camera IC butas coordinates (X, Y, Z) of the imaging area of the virtual camera ICcorresponding to the predetermined area T. The following descriptionwill be given with an imaging direction (rH, rV) and an angle of view(a) of the virtual camera IC.

The relationship between the predetermined area information and theimage of the predetermined area T will be described with FIG. 8.

FIG. 8 is a diagram illustrating the relationship between thepredetermined area information and the image of the predetermined areaT. As illustrated in FIG. 8, rH represents the horizontal radian, and rVrepresents the vertical radian. Further, a represents the angle of view.That is, the attitude of the virtual camera IC is changed such that thepoint of interest of the virtual camera IC represented by the imagingdirection (rH, rV) corresponds to a center point CP of the predeterminedarea T as the imaging area of the virtual camera IC. The predeterminedarea image Q is the image of the predetermined area T of theomnidirectional image CE in FIG. 6. Further, f represents the distancefrom the virtual camera IC to the center point CP, and L represents thedistance between a given vertex of the predetermined area T and thecenter point CP. Thus, 2L represents the length of a diagonal of thepredetermined area T. Further, in FIG. 8, a trigonometric functiontypically expressed as L/f=tan(α/2) holds.

FIG. 9 is a diagram illustrating a point in a three-dimensionalEuclidean space represented by spherical coordinates. The positioncoordinates of the center point CP are expressed as (r, θ, φ) in aspherical polar coordinate system. Herein, r, θ, and φ represent theradius vector, the polar angle, and the azimuth, respectively. Theradius vector r corresponds to the distance from the center point CP tothe origin of a three-dimensional virtual space including theomnidirectional image, and thus is equal to the distance f. FIG. 9illustrates the relationships between these elements. In the followingdescription, the center point CP will be described with the positioncoordinates (r, θ, φ) thereof.

A schematic configuration of the image communication system 10 of theembodiment will be described with FIG. 10.

FIG. 10 is a schematic diagram illustrating a configuration of the imagecommunication system 10 of the embodiment. As illustrated in FIG. 10, inthe image communication system 10 of the embodiment, communicationterminals 1, 2, 3 and 4 placed at sites A, B, C and D communicate witheach other via the communication network 100 such as the Internet toshare images therebetween. The image capturing device 5 a and thecommunication terminal 1 are placed at the site A, and the communicationterminal 2 and an image capturing device 5 b are placed at the site B.Further, an image capturing device 5 c, the communication terminal 3,and a display 6 c are placed at the site C, and the communicationterminal 4 and a display 6 d are placed at the site D.

Each of the image capturing devices 5 a and 5 c is a special digitalcamera for capturing the image of a subject (e.g., an object orsurroundings) and obtaining two hemispherical images to generate anomnidirectional image based on the hemispherical images. The imagecapturing device 5 b, on the other hand, is a commonly used digitalcamera for capturing the image of a subject (e.g., an object orsurroundings) and obtaining a typical planar image.

The communication terminals 3 and 4 are video conference terminalsdedicated to video conference. The communication terminals 3 and 4display the image of a video call on the displays 6 c and 6 d,respectively, via a wired cable such as a universal serial bus (USB)cable. Each of the communication terminals 3 and 4 normally captures theimage of a user, for example, with a camera 312 (see FIG. 12). Forexample, however, if the communication terminal 3 is connected, via awired cable, to a cradle 7 to which the image capturing device 5 c isattached, the image capturing device 5 c is given priority over thecamera 312 to obtain the omnidirectional image. When used with the wiredcable, the cradle 7 not only enables communication between the imagecapturing device 5 c and the communication terminal 3 but also supportsthe image capturing device 5 c and supplies power thereto.

Each of the communication terminals 1 and 2 is a general-purposeinformation processing apparatus that communicates with anothercommunication terminal at another site by operating application softwarefor video conference. The communication terminal 1 may be, but is notlimited to, a laptop personal computer (PC), a mobile phone, asmartphone, a tablet terminal, a car navigation system, a game console,a personal digital assistant (PDA), a wearable PC, or a desktop PC, forexample.

The communication terminal 1 displays the image of a video call on adisplay 917 (see FIG. 14) of the communication terminal 1. Thecommunication terminal 1 normally captures the image with acomplementary metal oxide semiconductor (CMOS) sensor 905 (see FIG. 14)of the communication terminal 1. With a wireless communicationtechnology conforming to a standard such as the Wi-Fi or Bluetooth(registered trademark) standard, the communication terminal 1 is alsocapable of acquiring the omnidirectional image obtained by the imagecapturing device 5 a.

The communication terminal 2 displays the image of a video call on thedisplay 917 of the communication terminal 2. The communication terminal2 captures the image of a user, for example, with image capturing device5 b externally attached to the communication terminal 2.

Each of the communication terminals 1 to 4 is installed with OpenGL ESto generate the predetermined area information representing thepredetermined area T forming a part of the omnidirectional image orgenerate the predetermined area image from the omnidirectional imagetransmitted from another communication terminal. The communicationterminals 1 to 4 are therefore capable of displaying the predeterminedarea image cut out from the omnidirectional image.

At the site A, there is at least one photographer 8 holding in a handthe image capturing device 5 a or a stick member attached to the imagecapturing device 5 a. The photographer 8 is able to move with the imagecapturing device 5 a. The communication terminal 1 displays on thedisplay 917 thereof the images captured by the image capturing devices 5a to 5 c and the camera 312 of the communication terminal 4.

At the site B, there is one user 9 b. The communication terminal 2displays on the display 917 thereof the images captured by the imagecapturing devices 5 a to 5 c and the camera 312 of the communicationterminal 4. The user 9 b at the site B is included in users who payattention to the point of interest specified by the photographer 8. Thenumber of users at the site B, which is one in this example, isillustrative.

At the site C, there are two users 9 c 1 and 9 c 2. The communicationterminal 3 displays on the display 6 c the images captured by the imagecapturing devices 5 a to 5 c and the camera 312 of the communicationterminal 4. The users 9 c 1 and 9 c 2 at the site C are included in theusers who pay attention to the point of interest specified by thephotographer 8. The number of users at the site C, which is two in thisexample, is illustrative.

At the site D, there is one user 9 d. The communication terminal 4displays on the display 6 d the images captured by the image capturingdevices 5 a to 5 c and the camera 312 of the communication terminal 4.The user 9 d at the site D is included in the users who pay attention tothe point of interest specified by the photographer 8. The number ofusers at the site D, which is one in this example, is illustrative.

A communication management system 50 manages and controls thecommunication between the communication terminals 1 to 4, and thus alsofunctions as a communication control system. The communicationmanagement system 50 is installed in, for example, a service providercompany that provides a video communication service. The communicationmanagement system 50 may be implemented by a single computer, or may beimplemented by a plurality of computers to which units (e.g., functions,devices, and memories) of the communication management system 50 aredivided and allocated.

The number of sites, the type of the communication terminals 1 to 4placed at the respective sites, the type of the image capturing devices5 a to 5 c, and the number of users illustrated in FIG. 10 areillustrative. In the present embodiment, the minimum number of sites istwo: the site A and another site. At the site B, the image capturingdevice 5 b may be omitted; it suffices if the communication terminal 2is capable of displaying the omnidirectional image transmitted from thesite A.

Further, the image capturing device 5 a and the communication terminal 1at the site A may be integrated together. That is, if the imagecapturing device 5 a has a function to connect to the communicationnetwork 100, the communication terminal 1 may be omitted. In this case,the image capturing device 5 a functions as the communication terminal1. This is because, in the present embodiment, the photographer 8 isable to specify the point of interest with the image capturing device 5a. If the communication terminal 1 is provided at the site A, however,the photographer 8 is also able to specify the point of interest byoperating the communication terminal 1. Further, the communicationterminal 1 may receive the omnidirectional image and the coordinates ofthe point of interest via a storage medium, without communicating withthe image capturing device 5 a at the site A.

In the following description, a given one of the image capturing devices5 a to 5 c will be simply described as the image capturing device 5.Similarly, a given one of the displays 6 c and 6 d will be simplydescribed as the display 6.

With reference to FIG. 11 to FIG. 14, a detailed description will begiven of respective hardware configurations of the image capturingdevices 5 a and 5 c, the communication terminals 1 to 4, and thecommunication management system 50 of the embodiment. The imagecapturing device 5 b is a commonly used camera, and thus descriptionthereof will be omitted.

A hardware configuration of each of the image capturing devices 5 a and5 c will be described with FIG. 11.

FIG. 11 is a diagram illustrating a hardware configuration of each ofthe image capturing devices 5 a and 5 c. The following description willbe given on the assumption that each of the image capturing devices 5 aand 5 c is an omnidirectional (i.e., all-directional) image capturingdevice with two imaging elements. However, the number of imagingelements may be three or more. Further, each of the image capturingdevices 5 a and 5 c is not necessarily required to be a device dedicatedto the purpose of capturing the all-directional image. Therefore, anall-directional image capturing device may be additionally attached to atypical digital camera or smartphone, for example, to providesubstantially the same functions as those of the image capturing device5 a or 5 c.

As illustrated in FIG. 11, each of the image capturing devices 5 a and 5c includes an imaging device 101, an image processing device 104, animaging control device 105, a microphone 108, an audio processing device109, a central processing unit (CPU) 111, a read only memory (ROM) 112,a static random access memory (SRAM) 113, a dynamic RAM (DRAM) 114, anoperation device 115, a network interface (I/F) 116, a communicationdevice 117, an antenna 117 a, an acceleration and orientation sensor118, and a gyro sensor 119.

The imaging device 101 includes two fisheye lenses 102 a and 102 b andtwo imaging elements 103 a and 103 b corresponding thereto. The fisheyelenses 102 a and 102 b are wide-angle lenses each having an angle ofview of at least 180 degrees for forming a hemispherical image. Each ofthe imaging elements 103 a and 103 b includes an image sensor, a timinggenerating circuit, and a group of registers, for example. For example,the image sensor may be a CMOS or charge coupled device (CCD) sensorthat converts an optical image formed by the fisheye lens 102 a or 102 binto image data of electrical signals and outputs the image data. Thetiming generating circuit generates a horizontal or verticalsynchronization signal or a pixel clock signal for the image sensor.Various commands and parameters for the operation of the imaging element103 a or 103 b are set in the group of registers.

Each of the imaging elements 103 a and 103 b of the imaging device 101is connected to the image processing device 104 via a parallel I/F bus,and is connected to the imaging control device 105 via a serial I/F bus(e.g., an inter-integrated circuit (I²C) bus). The image processingdevice 104 and the imaging control device 105 are connected to the CPU111 via a bus 110. The bus 110 is further connected to the ROM 112, theSRAM 113, the DRAM 114, the operation device 115, the network I/F 116,the communication device 117, the acceleration and orientation sensor118, and the gyro sensor 119, for example.

The image processing device 104 receives image data items from theimaging elements 103 a and 103 b via the parallel I/F bus, performs apredetermined process on the image data items, and combines theprocessed image data items to generate the data of the equidistantcylindrical image as illustrated in FIG. 4C.

The imaging control device 105 sets commands in the groups of registersof the imaging elements 103 a and 103 b via the serial I/F bus such asthe I²C bus, with the imaging control device 105 and imaging elements103 a and 103 b acting as a master device and slave devices,respectively. The imaging control device 105 receives the commands fromthe CPU 111. The imaging control device 105 further receives data suchas status data from the groups of registers of the imaging elements 103a and 103 b via the serial I/F bus such as the I²C bus, and transmitsthe received data to the CPU 111.

The imaging control device 105 further instructs the imaging elements103 a and 103 b to output the image data when the shutter button SB ofthe operation device 115 is pressed down. The image capturing device 5 aor 5 c may have a preview display function or a video display functionusing a display (e.g., the display 917 of the communication terminal 1or the display 6 c connected to the communication terminal 3). In thiscase, the imaging elements 103 a and 103 b continuously output the imagedata at a predetermined frame rate. The frame rate is defined as thenumber of frames per minute.

The imaging control device 105 also functions as a synchronizationcontroller that cooperates with the CPU 111 to synchronize the imagedata output time between the imaging elements 103 a and 103 b. In thepresent embodiment, the image capturing devices 5 a and 5 c are notequipped with a display, but may be equipped with a display.

The microphone 108 converts sound into audio (signal) data. The audioprocessing device 109 receives the audio data from the microphone 108via an I/F bus, and performs a predetermined process on the audio data.

The CPU 111 controls an overall operation of the image capturing device5 a or 5 c, and executes various processes. The ROM 112 stores variousprograms for the CPU 111. The SRAM 113 and the DRAM 114 are used as workmemories, and store programs executed by the CPU 111 and data beingprocessed. The DRAM 114 particularly stores image data being processedby the image processing device 104 and processed data of the equidistantcylindrical image.

The operation device 115 collectively refers to components such asvarious operation buttons including the shutter button SB, a powerswitch, and a touch panel that has a display function and an operationfunction. The user operates the operation buttons to input variousimaging modes and imaging conditions, for example.

The network I/F 116 collectively refers to interface circuits (e.g., aUSB I/F circuit) to connect to external devices or apparatuses such asan external medium (e.g., a secure digital (SD) card) and a PC. Thenetwork I/F 116 may be a wireless or wired interface. Via the networkI/F 116, the data of the equidistant cylindrical image stored in theDRAM 114 may be recorded on an external medium, or may be transmitted asnecessary to an external apparatus such as the communication terminal 3.

The communication device 117 communicates with an external apparatussuch as the communication terminal 1 or 3 via the antenna 117 a of theimage capturing device 5 a or 5 c in accordance with a near fieldwireless communication technology conforming to the Wi-Fi or near fieldcommunication (NFC) standard, for example. The data of the equidistantcylindrical image may also be transmitted to an external apparatus suchas the communication terminal 1 or 3 via the communication device 117.

The acceleration and orientation sensor 118 outputs orientation and tiltinformation by calculating the orientation and tilt (i.e., the rollangle, the yaw angle, and the pitch angle) of the image capturing device5 a or 5 c from the geomagnetism. The orientation and tilt informationis an example of related information (i.e., meta data) conforming to theexchangeable image file format (Exif) standard. The orientation and tiltinformation is used in image processing such as image correction of thecaptured image. The related information includes data such as the dateand time of capturing the image and the data capacity of the image data.

The gyro sensor 119 is a triaxial or hexaxial sensor that detects therespective rates of rotation around the X-axis, the Y-axis, and theZ-axis. The rates of rotation are accumulated (i.e., integrated) toobtain respective rotation angles, which represents the attitudeinformation.

A hardware configuration of each of the communication terminals 3 and 4(i.e., the video conference terminal) will be described with FIG. 12.

FIG. 12 is a diagram illustrating a hardware configuration of each ofthe communication terminals 3 and 4. As illustrated in FIG. 12, each ofthe communication terminals 3 and 4 includes a CPU 301, a ROM 302, a RAM303, a flash memory 304, a solid state drive (SSD) 305, a medium I/F307, operation buttons 308, a power switch 309, a bus line 310, anetwork I/F 311, a camera 312, an imaging element I/F 313, a microphone314, a speaker 315, an audio input and output I/F 316, a display I/F317, an external apparatus connection I/F 318, a near fieldcommunication circuit 319, and an antenna 319 a for the near fieldcommunication circuit 319.

The CPU 301 controls an overall operation of the communication terminal3 or 4. The ROM 302 stores a program used to drive the CPU 301 such asan initial program loader (IPL). The RAM 303 is used as a work area forthe CPU 301. The flash memory 304 stores a communication program, imagedata, audio data, and other various data. The SSD 305 controls writingand reading of various data to and from the flash memory 304 under thecontrol of the CPU 301. The SSD 305 may be replaced by a hard disk drive(HDD). The medium I/F 307 controls writing (i.e., storage) and readingof data to and from a recording medium 306 such as a flash memory. Theoperation buttons 308 are operated in the selection of an address by thecommunication terminal 3 or 4, for example. The power switch 309 is usedto switch between ON and OFF of power supply to the communicationterminal 3 or 4.

The network I/F 311 is an interface for data communication via thecommunication network 100 such as the Internet. The camera 312 is abuilt-in image capturing device that captures the image of a subjectunder the control of the CPU 301 to obtain image data. The imagingelement I/F 313 is a circuit that controls driving of the camera 312.The microphone 314 is a built-in sound collecting device for inputtingsounds. The audio input and output I/F 316 is a circuit that processesinput of audio signals from the microphone 314 and output of audiosignals to the speaker 315 under the control of the CPU 301. The displayI/F 317 is a circuit that transmits the image data to the externaldisplay 6 under the control of the CPU 301. The external apparatusconnection I/F 318 is an interface for connecting the communicationterminal 3 or 4 to various external apparatuses. The near fieldcommunication circuit 319 is a communication circuit conforming to theNFC or Bluetooth standard, for example.

The bus line 310 includes an address bus and a data bus for electricallyconnecting the components of the communication terminal 3 or 4illustrated in FIG. 12 including the CPU 301.

The display 6 is implemented as a liquid crystal or organic electroluminescence (EL) display, for example, that displays the image of thesubject and icons for performing various operations. The display 6 isconnected to the display I/F 317 by a cable 6 y. The cable 6 y may be acable for analog red, green, blue (RGB) video graphics array (VGA)signals, a cable for component video, or a cable for high-definitionmultimedia interface (HDMI, registered trademark) or digital videointeractive (DVI) signals.

The camera 312 includes lenses and a solid-state image sensing devicethat converts light into electric charge to digitize the still or videoimage of the subject. The solid-state image sensing device is a CMOS orCCD sensor, for example. The external apparatus connection I/F 318 isconnectable to an external apparatus such as an external camera, anexternal microphone, or an external speaker via a USB cable, forexample. If an external camera is connected to the external apparatusconnection I/F 318, the external camera is driven in preference to thebuilt-in camera 312 under the control of the CPU 301. Similarly, if anexternal microphone or speaker is connected to the external apparatusconnection I/F 318, the external microphone or speaker is driven inpreference to the built-in microphone 314 or speaker 315 under thecontrol of the CPU 301.

The recording medium 306 is removable from the communication terminal 3or 4. The recording medium 306 may be a nonvolatile memory capable ofreading and writing data under the control of the CPU 301. In this case,the flash memory 304 may be replaced by an electrically erasable andprogrammable ROM (EEPROM), for example.

A hardware configuration of the communication management system 50 willbe described with FIG. 13.

FIG. 13 is a diagram illustrating a hardware configuration of thecommunication management system 50. The communication management system50 includes a CPU 501, a ROM 502, a RAM 503, a hard disk (HD) 504, anHDD 505, a medium I/F 507, a display 508, a network I/F 509, a keyboard511, a mouse 512, a compact disc-rewritable (CD-RW) drive 514, and a busline 510.

The CPU 501 controls an overall operation of the communicationmanagement system 50. The ROM 502 stores a program used to drive the CPU501 such as an IPL. The RAM 503 is used as a work area for the CPU 501.The HD 504 stores various data of a program for the communicationmanagement system 50, for example. The HDD 505 controls writing andreading of various data to and from the HD 504 under the control of theCPU 501. The medium I/F 507 controls writing (i.e., storage) and readingof data to and from a recording medium 506 such as a flash memory. Thedisplay 508 displays various information such as a cursor, menus,windows, text, and images. The network I/F 509 is an interface for datacommunication via the communication network 100. The keyboard 511includes a plurality of keys for inputting text, numerical values, andvarious instructions, for example. The mouse 512 is used to select andexecute various instructions, select a processing target, and move thecursor, for example. The CD-RW drive 514 controls reading of variousdata from a CD-RW 513 as an example of a removable recording medium. Thebus line 510 includes an address bus and a data bus for electricallyconnecting the above-described components of the communicationmanagement system 50, as illustrated in FIG. 13.

A hardware configuration of each of the communication terminals 1 and 2will be described with FIG. 14.

FIG. 14 is a diagram illustrating a hardware configuration of each ofthe communication terminals 1 and 2. As illustrated in FIG. 14, each ofthe communication terminals 1 and 2 includes a CPU 901, a ROM 902, a RAM903, an EEPROM 904, a CMOS sensor 905, an acceleration and orientationsensor 906, a medium I/F 908, and a global positioning system (GPS)receiver 909.

The CPU 901 controls an overall operation of the communication terminal1 or 2. The ROM 902 stores a program used to drive the CPU 901 such asan IPL. The RAM 903 is used as a work area for the CPU 901. The EEPROM904 performs reading and writing of various data of a program for thecommunication terminal 1 or 2, for example, under the control of the CPU901. The CMOS sensor 905 captures the image of a subject (normally theimage of a user) under the control of the CPU 901 to obtain image data.The acceleration and orientation sensor 906 includes various sensorssuch as an electromagnetic compass that detects geomagnetism, agyrocompass, and an acceleration sensor. The medium I/F 908 controlswriting (i.e., storage) and reading of data to and from a recordingmedium 907 such as a flash memory. The GPS receiver 909 receives a GPSsignal from a GPS satellite.

Each of the communication terminals 1 and 2 further includes atelecommunication circuit 911, an antenna 911 a for thetelecommunication circuit 911, a camera 912, an imaging element I/F 913,a microphone 914, a speaker 915, an audio input and output I/F 916, adisplay 917, an external apparatus connection I/F 918, a near fieldcommunication circuit 919, an antenna 919 a for the near fieldcommunication circuit 919, a touch panel 921, and a bus line 910.

The telecommunication circuit 911 is a circuit that communicates withanother apparatus via the communication network 100. The camera 912 is abuilt-in image capturing device that captures the image of a subjectunder the control of the CPU 901 to obtain image data. The imagingelement I/F 913 is a circuit that controls driving of the camera 912.The microphone 914 is a built-in sound collecting device for inputtingsounds. The audio input and output I/F 916 is a circuit that thatprocesses input of audio signals from the microphone 914 and output ofaudio signals to the speaker 915 under the control of the CPU 901. Thedisplay 917 is implemented as a liquid crystal or organic EL display,for example, that displays the image of the subject and various icons.The external apparatus connection I/F 918 is an interface for connectingthe communication terminal 1 or 2 to various external apparatuses. Thenear field communication circuit 919 is a communication circuitconforming to the NFC or Bluetooth standard, for example. The touchpanel 921 is an input device for the user to operate the communicationterminal 1 or 2 by pressing the display 917. The bus line 910 includesan address bus and a data bus for electrically connecting theabove-described components of the communication terminal 1 or 2including the CPU 901.

A recording medium (e.g., a CD-ROM) or an HD storing the above-describedprograms may be shipped to the market as a program product.

A functional configuration of the image communication system 10 of thepresent embodiment will be described with FIGS. 15A to 22.

FIGS. 15A to 16B are functional block diagrams illustrating functionalblocks of the image communication system 10. As illustrated in FIG. 15A,the image capturing device 5 a includes a receiving unit 12 a, animaging unit 13 a, a sound collecting unit 14 a, an attitude informationacquiring unit 15 a, an image processing unit 16 a, a communication unit18 a, and a storing and reading unit 19 a. Except for the imageprocessing unit 16 a, each of these units is a function or deviceimplemented when at least one of the components illustrated in FIG. 11operates in response to a command from the CPU 111 in accordance with aprogram for the image capturing device 5 a deployed on the DRAM 114 fromthe SRAM 113. The image processing unit 16 a is implemented by a circuitmodule such as an application specific integrated circuit (ASIC), adigital signal processor (DSP), or a field programmable gate array(FPGA). Alternatively, the image processing unit 16 a may be implementedby software.

The image capturing device 5 a further includes a storage unit 1000 aimplemented by the ROM 112, the SRAM 113, and the DRAM 114 illustratedin FIG. 11. The storage unit 1000 a stores globally unique identifiers(GUIDs) of the image capturing device 5 a.

In the image capturing device 5 a, the receiving unit 12 a is mainlyimplemented by the operation device 115 and the processing of the CPU111 illustrated in FIG. 11. The receiving unit 12 a receives anoperation input from the photographer 8.

The imaging unit 13 a is mainly implemented by the imaging device 101,the image processing device 104, the imaging control device 105, and theprocessing of the CPU 111 illustrated in FIG. 11. The imaging unit 13 acaptures the image of surroundings of the image capturing device 5 a,for example, to obtain captured image data (i.e., the omnidirectionalimage).

The sound collecting unit 14 a is implemented by the microphone 108, theaudio processing device 109, and the processing of the CPU 111illustrated in FIG. 11. The sound collecting unit 14 a collects soundsof the surroundings of the image capturing device 5 a.

The attitude information acquiring unit 15 a acquires the attitudeinformation of the attitude of the image capturing device 5 a (i.e., thetilt of the image capturing device 5 a in a three-dimensional space,which may be represented by components of roll, pitch, and yaw). Theimage processing unit 16 a executes a process of combining twohemispherical images to generate one equidistant cylindrical image. Thisprocess includes the zenith correction based on the attitudeinformation. Thereby, the coordinates of the point of interest pointedby the upper side in the longitudinal direction of the image capturingdevice 5 a are converted into spherical coordinates (i.e., coordinatesof the equidistant cylindrical image) indicating the direction pointedby the photographer 8.

The communication unit 18 a is mainly implemented by the processing ofthe CPU 111 illustrated in FIG. 11. The communication unit 18 acommunicates with a later-described communication unit 98 of thecommunication terminal 1 in accordance with a near field wirelesscommunication technology conforming to the NFC, Bluetooth, or Wi-Fistandard, for example.

The storing and reading unit 19 a is mainly implemented by theprocessing of the CPU 111 illustrated in FIG. 11. The storing andreading unit 19 a stores various data and information in the storageunit 1000 a, and reads therefrom various data and information.

The image capturing device 5 c includes a receiving unit 12 c, animaging unit 13 c, a sound collecting unit 14 c, an attitude informationacquiring unit 15 c, an image processing unit 16 c, a communication unit18 c, a storing and reading unit 19 c, and a storage unit 1000 c. Theseunits implement similar functions to those of the receiving unit 12 a,the imaging unit 13 a, the sound collecting unit 14 a, the attitudeinformation acquiring unit 15 a, the image processing unit 16 a, thecommunication unit 18 a, the storing and reading unit 19 a, and thestorage unit 1000 a of the image capturing device 5 a, and thusdescription thereof will be omitted.

A functional configuration of the communication terminal 1 will bedescribed.

As illustrated in FIG. 15B, the communication terminal 1 includes atransmitting and receiving unit 91, a receiving unit 92, an image andaudio processing unit 93, a display control unit 94, a determinationunit 95, a generation unit 96, a calculation unit 97, a communicationunit 98, and a storing and reading unit 99. Each of these units is afunction or device implemented when at least one of the componentsillustrated in FIG. 14 operates in response to a command from the CPU901 in accordance with the program for the communication terminal 1deployed on the RAM 903 from the EEPROM 904.

The communication terminal 1 further includes a storage unit 9000implemented by the ROM 902, the RAM 903, and the EEPROM 904 illustratedin FIG. 14. The storage unit 9000 stores an image type managementdatabase (DB) 9001, an image capturing device management DB 9002, and apredetermined area management DB 9003.

The image type management DB 9001 is configured as an image typemanagement table illustrated in FIG. 17. The image capturing devicemanagement DB 9002 is configured as an image capturing device managementtable illustrated in FIG. 18. The predetermined area management DB 9003is configured as a predetermined area information management tableillustrated in FIG. 19.

FIG. 17 is a conceptual diagram illustrating the image type managementtable. In the image type management table, an image data ID, an internetprotocol (IP) address, and a source name are stored and managed inassociation with each other. The image data ID is an example of imagedata identification information for identifying the image data in videocommunication. Image data items transmitted from the same transmitterterminal are assigned with the same image data ID, enabling adestination terminal (i.e., receiver terminal) to identify thetransmitter terminal having transmitted the image data received by thereceiver terminal. The IP address is an example of the address of thetransmitter terminal. The IP address of the transmitter terminalrepresents the IP address of the communication terminal that transmitsthe image data represented by the image data ID associated with the IPaddress. The source name is a name for identifying the image capturingdevice that outputs the image data represented by the image data IDassociated with the source name. The source name is an example of imagetype information. The source name is generated by the communicationterminal 3, for example, in accordance with a predetermined naming rule.

For example, the image type management table of FIG. 17 indicates thatfour communication terminals with respective IP addresses “1.2.1.3,”“1.2.2.3,” “1.3.1.3,” and “1.3.2.3” have transmitted image data itemsrepresented by image data IDs “RS001,” “RS002,” “RS003,” and“RS004.” Theimage type management table further indicates that image typesrepresented by respective source names of the communication terminalsare “Video_Wide,” “Video_Wide,” “Video,” and “Video,” which represent“special image,” “special image,” “general image,” and “general image,”respectively. Herein, the special image is the omnidirectional image.

Data other than the image data may also be managed in association withthe image data ID. For example, the data other than the image dataincludes audio data and material data used in sharing the imagedisplayed on the screen.

FIG. 18 is a conceptual diagram illustrating the image capturing devicemanagement table. In the image capturing device management table, avendor ID (VID) and a product ID (PID) are stored and managed. The VIDand the PID are included in the GUIDs of an image capturing devicecapable of capturing two hemispherical images, based on which theomnidirectional image is generated. For example, the VID and the PIDused by a USB device may be used as the GUIDs. The VID and the PID arestored in the communication terminals 3 and 4, for example, in factoryshipment thereof, but may be additionally stored therein after thefactory shipment.

FIG. 19 is a conceptual diagram illustrating the predetermined areainformation management table. In the predetermined area informationmanagement table, the IP address of the transmitter terminal, the IPaddress of the destination terminal, and the predetermined areainformation are stored and managed in association with each other. TheIP address of the transmitter terminal is the IP address of thecommunication terminal that transmits the captured image data. The IPaddress of the destination terminal is the IP address of thecommunication terminal to which the captured image data is transmitted.The predetermined area information represents the predetermined areaimage being displayed by the destination terminal, i.e., thecommunication terminal to which the captured image data is transmitted.The destination terminal to which the captured image data is transmittedis also the transmitter terminal that transmits the predetermined areainformation. The predetermined area information is configured as aconversion table for converting the captured image into thepredetermined area image, i.e., the image of the predetermined area T ofthe captured image, as illustrated in FIGS. 6 to 7B. The IP address isan example of address information. The address information includes amedia access control (MAC) address and a terminal identification (ID)for identifying the communication terminal. The IP address illustratedin FIG. 19 is a simplified version of the internet protocol version(IPv)4 address. Alternatively, the IPv6 address may be used as the IPaddress.

For example, when the IP address of the communication terminal 3 is“1.2.1.3,” the information managed in the first to third rows of thepredetermined area information management table in FIG. 19 indicatesthat the captured image data transmitted from the communication terminal3 has been transmitted, via the communication management system 50, tothe communication terminal 4 corresponding to the IP address “1.2.2.3,”the communication terminal 2 corresponding to the IP address “1.3.1.3,”and the communication terminal 1 corresponding to the IP address“1.3.2.3.” The information further indicates that the communicationterminal 4 is a transmitter terminal having transmitted predeterminedarea information (r=10, θ=20, φ=30), that the communication terminal 2is a transmitter terminal having transmitted predetermined areainformation (r=20, θ=30, φ=40), and that the communication terminal 1 isa transmitter terminal having transmitted predetermined area information(r=30, θ=40, φ=50). That is, the predetermined area informationmanagement table manages the predetermined area information used by thecommunication terminals 1, 2, and 4 to display the omnidirectionalimage.

If the transmitting and receiving unit 91 receives new predeterminedarea information corresponding to the IP address of the transmitterterminal of already-managed captured image data and the IP address ofthe destination terminal of the already-managed captured image data, thestoring and reading unit 99 overwrites the corresponding already-managedpredetermined area information with the newly received predeterminedarea information.

The transmitting and receiving unit 91 of the communication terminal 1is mainly implemented by the telecommunication circuit 911 and theprocessing of the CPU 901 illustrated in FIG. 14. The transmitting andreceiving unit 91 transmits and receives various data and information toand from the communication management system 50 via the communicationnetwork 100.

The receiving unit 92 is mainly implemented by the touch panel 921 andthe processing of the CPU 901. The receiving unit 92 receives variousselections and inputs from the user. As well as the touch panel 921, thereceiving unit 92 may also be implemented by another input device suchas an audio input device.

The image and audio processing unit 93 is implemented by a command fromthe CPU 901 illustrated in FIG. 14. The image and audio processing unit93 performs image processing on the image data of the image of thesubject captured by the camera 912. The image and audio processing unit93 further performs audio processing on audio data of audio signalsconverted from the voice of the user by the microphone 914.

Based on the image type information such as the source name, the imageand audio processing unit 93 performs image processing on the image datareceived from another communication terminal such that the displaycontrol unit 94 displays a resultant image on the display 917. The imageand audio processing unit 93 further outputs, to the speaker 915, audiosignals of audio data received from another communication terminal viathe communication management system 50, to thereby output sound from thespeaker 915.

The display control unit 94 is mainly implemented by the display 917 andthe processing of the CPU 901. The display control unit 94 executescontrol for displaying various images and text on the display 917.

The determination unit 95 is mainly implemented by the processing of theCPU 901. The determination unit 95 determines the image type of theimage data received from the image capturing device 5 a, for example.

The generation unit 96 is mainly implemented by the processing of theCPU 901. Based on the determination by the determination unit 95 ofwhether the image type of the image data is the general image or thespecial image (i.e., the omnidirectional image in the present example),the generation unit 96 generates the source name (an example of theimage type information) in accordance with the foregoing naming rule.For example, if the determination unit 95 determines that the image typeis the general image, the generation unit 96 generates a source name“Video” indicating that the image type is the general image. Further, ifthe determination unit 95 determines that the image type is the specialimage, the generation unit 96 generates a source name “Video_Wide”indicating that the image type is the special image.

The calculation unit 97 is mainly implemented by the processing of theCPU 901. Based on the predetermined area information representing thepredetermined area T and the predetermined area information receivedfrom another communication terminal by the transmitting and receivingunit 91, the calculation unit 97 calculates the position (i.e., positioninformation) of the predetermined area T in the captured image. When theentirety of the captured image is displayed, the entire captured imagewill also be referred to as the whole image.

The communication unit 98 is mainly implemented by the near fieldcommunication circuit 919, the antenna 919 a, and the processing of theCPU 901. The communication unit 98 communicates with the communicationunit 18 a of the image capturing device 5 a in accordance with a nearfield wireless communication technology conforming to the NFC,Bluetooth, or Wi-Fi standard, for example. In the present example, thecommunication unit 98 and the transmitting and receiving unit 91 areconfigured as separate communication units, but may be integratedtogether.

The storing and reading unit 99 is mainly implemented by the processingof the CPU 901. The storing and reading unit 99 stores various data andinformation in the storage unit 9000, and reads therefrom various dataand information.

A functional configuration of the communication terminal 2 will bedescribed with FIG. 16B.

The communication terminal 2 has basically the same functions as thoseof the communication terminal 1. That is, as illustrated in FIG. 16B,the communication terminal 2 includes a transmitting and receiving unit71, a receiving unit 72, an image and audio processing unit 73, adisplay control unit 74, a determination unit 75, a generation unit 76,a calculation unit 77, a communication unit 78, and a storing andreading unit 79. These units implement similar functions to those of thetransmitting and receiving unit 91, the receiving unit 92, the image andaudio processing unit 93, the display control unit 94, the determinationunit 95, the generation unit 96, the calculation unit 97, thecommunication unit 98, and the storing and reading unit 99 of thecommunication terminal 1, and thus description thereof will be omitted.

The communication terminal 2 further includes a storage unit 7000implemented by the ROM 902, the RAM 903, and the EEPROM 904 illustratedin FIG. 14. The storage unit 7000 stores an image type management DB7001, an image capturing device management DB 7002, and a predeterminedarea management DB 7003. These databases are the same in data structureas the image type management DB 9001, the image capturing devicemanagement DB 9002, and the predetermined area management DB 9003 of thecommunication terminal 1, and thus description thereof will be omitted.

A functional configuration of the communication terminal 3 will bedescribed with FIG. 15B.

The communication terminal 3 has basically the same functions as thoseof the communication terminal 1. That is, as illustrated in FIG. 15B,the communication terminal 3 includes a transmitting and receiving unit31 c, a receiving unit 32 c, an image and audio processing unit 33 c, adisplay control unit 34 c, a determination unit 35 c, a generation unit36 c, a calculation unit 37 c, a communication unit 38 c, and a storingand reading unit 39 c. These units implement similar functions to thoseof the transmitting and receiving unit 91, the receiving unit 92, theimage and audio processing unit 93, the display control unit 94, thedetermination unit 95, the generation unit 96, the calculation unit 97,the communication unit 98, and the storing and reading unit 99 of thecommunication terminal 1, and thus description thereof will be omitted.

The communication terminal 3 further includes a storage unit 3000 cimplemented by the ROM 302, the RAM 303, and the SSD 305 illustrated inFIG. 12. The storage unit 3000 c stores an image type management DB 3001c, an image capturing device management DB 3002 c, and a predeterminedarea management DB 3003 c. These databases implement similar functionsto those of the image type management DB 9001, the image capturingdevice management DB 9002, and the predetermined area management DB 9003of the communication terminal 1, and thus description thereof will beomitted.

A functional configuration of the communication terminal 4 will bedescribed with FIG. 16B.

The communication terminal 4 has basically the same functions as thoseof the communication terminal 1. That is, as illustrated in FIG. 16B,the communication terminal 4 includes a transmitting and receiving unit31 d, a receiving unit 32 d, an image and audio processing unit 33 d, adisplay control unit 34 d, a determination unit 35 d, a generation unit36 d, a calculation unit 37 d, a communication unit 38 d, and a storingand reading unit 39 d. These units implement similar functions to thoseof the transmitting and receiving unit 91, the receiving unit 92, theimage and audio processing unit 93, the display control unit 94, thedetermination unit 95, the generation unit 96, the calculation unit 97,the communication unit 98, and the storing and reading unit 99 of thecommunication terminal 1, and thus description thereof will be omitted.

The communication terminal 4 further includes a storage unit 3000 dimplemented by the ROM 302, the RAM 303, and the SSD 305 illustrated inFIG. 12. The storage unit 3000 d stores an image type management DB 3001d, an image capturing device management DB 3002 d, and a predeterminedarea management DB 3003 d. These databases implement similar functionsto those of the image type management DB 9001, the image capturingdevice management DB 9002, and the predetermined area management DB 9003of the communication terminal 1, and thus description thereof will beomitted.

A functional configuration of the communication management system 50will be described in detail with FIG. 16A.

The communication management system 50 includes a transmitting andreceiving unit 51, a determination unit 55, a generation unit 56, and astoring and reading unit 59. Each of these units is a function or deviceimplemented when at least one of the components illustrated in FIG. 13operates in response to a command from the CPU 501 in accordance withthe program for the communication management system 50 deployed on theRAM 503 from the HD 504.

The communication management system 50 further includes a storage unit5000 implemented by the RAM 503 and the HD 504 illustrated in FIG. 13.The storage unit 5000 stores a session management DB 5001, an image typemanagement DB 5002, and a predetermined area management DB 5003. Thesession management DB 5001 is configured as a session management tableillustrated in FIG. 20. The image type management DB 5002 is configuredas an image type management table illustrated in FIG. 21. Thepredetermined area management DB 5003 is configured as a predeterminedarea information management table illustrated in FIG. 22.

FIG. 20 is a conceptual diagram illustrating the session managementtable. In the session management table, a session ID and the IPaddresses of participant communication terminals are stored and managedin association with each other. The session ID is an example of sessionidentification information for identifying a communication session thatimplements a video call. The session ID is generated for each virtualmeeting room. The session ID is also managed in each of thecommunication terminals 1 to 4, and is used thereby in the selection ofa communication session. The IP addresses of the participantcommunication terminals represent the IP addresses of the communicationterminals participating in the communication session in the virtualmeeting room represented by the session ID associated with the IPaddresses.

FIG. 21 is a conceptual diagram illustrating the image type managementtable. In the image type management table of FIG. 21, in addition to theinformation items managed in the image type management table of FIG. 17,the session ID managed in the session management table is stored andmanaged in association with the information items. The image typemanagement table of FIG. 21 indicates that three communication terminalswith respective IP addresses “1.2.1.3,” “1.2.2.3,” and “1.3.1.3” areparticipating in the communication session in the virtual meeting roomrepresented by the same session ID “se101.” The communication managementsystem 50 manages the image data IDs, the IP addresses of thetransmitter terminals, and the source names (i.e., the image typeinformation) the same as those managed in communication terminals suchas the communication terminals 1 to 4 (i.e., video conferenceterminals). This is because, when a new communication terminal entersthe virtual meeting room, for example, the communication managementsystem 50 transmits the image type information and other information toboth the communication terminals already participating in the video calland the newly participating communication terminal. It is therebyunnecessary for the communication terminals already participating in thevideo call and the newly participating communication terminal totransmit and receive therebetween the image type information and otherinformation.

FIG. 22 is a conceptual diagram illustrating the predetermined areainformation management table. The predetermined area informationmanagement table of FIG. 22 is basically similar in data structure tothe predetermined area information management table of FIG. 19. Thetransmitting and receiving unit 51 transmits the latest predeterminedarea information to each of the communication terminals 1 to 4 atcertain time intervals (e.g., at every 30 seconds), as described later.During the transmission of the predetermined area information in each ofthe certain time intervals, therefore, all predetermined areainformation received by the transmitting and receiving unit 51 is savedwithout being deleted. In the predetermined area information managementtable of FIG. 22, the predetermined area information is managed suchthat a newer predetermined area information item is placed at a higherposition.

Referring back to FIG. 16A, the transmitting and receiving unit 51 ofthe communication management system 50 is mainly implemented by thenetwork I/F 509 and the processing of the CPU 501 illustrated in FIG.13. The transmitting and receiving unit 51 transmits and receivesvarious data and information to and from the communication terminals 1to 4 via the communication network 100.

The determination unit 55 is mainly implemented by the processing of theCPU 501, and makes various determinations.

The generation unit 56 is mainly implemented by the processing of theCPU 501, and generates the image data ID.

The storing and reading unit 59 is mainly implemented by the HDD 505 andthe processing of the CPU 501 illustrated in FIG. 13. The storing andreading unit 59 stores various data and information in the storage unit5000, and reads therefrom various data and information.

A process of having a communication terminal participate in a specificcommunication session will be described with FIGS. 23 and 24.

FIG. 23 is a sequence diagram illustrating a process of having acommunication terminal participate in a specific communication session.FIG. 24 is a diagram illustrating a selection screen for selecting acommunication session in a virtual meeting room.

The photographer 8 at the site A first operates the communicationterminal 1 to display the selection screen for selecting a communicationsession in a virtual meeting room. Then, in the communication terminal1, the receiving unit 92 receives the operation for displaying theselection screen, and the display control unit 94 displays the selectionscreen as illustrated in FIG. 24 on the display 917 of the communicationterminal 1 (step S21). The selection screen displays selection buttonsb1, b2, and b3, which represent virtual meeting rooms R1, R2, and R3,respectively, as options. The selection buttons b1, b2, and b3 areassociated with respective session IDs se101, se102, and se103.

The photographer 8 then selects one of the selection buttons b1, b2, andb3 corresponding to a desired virtual meeting room. It is assumed herethat the photographer 8 selects the selection button b1. Then, thereceiving unit 92 receives the selection of the communication session(step S22). The virtual meeting room selected here is one previouslydesignated for a certain purpose such as viewing of real estateproperties, for example.

Then, the transmitting and receiving unit 91 transmits a participationrequest to the communication management system 50 to participate in thecommunication session in the virtual meeting room (step S23). Theparticipation request includes the session ID representing thecommunication session, the selection of which has been received at stepS22, and the IP address of the communication terminal 1 transmitting theparticipation request. Then, the transmitting and receiving unit 51 ofthe communication management system 50 receives the participationrequest.

The storing and reading unit 59 of the communication management system50 then adds the IP address received at step S23 to the sessionmanagement DB 5001, specifically to the field of a participant terminalIP address in a record corresponding to the session ID received at stepS23. Thereby, a communication session participation process is performed(step S24).

The transmitting and receiving unit 51 then transmits a response to theparticipation request to the communication terminal 1 (step S25). Theresponse to the participation request includes the session ID receivedat step S23 and a result of the participation process. Then, thetransmitting and receiving unit 91 of the communication terminal 1receives the response to the participation request.

A procedure of a process following a successful participation processwill be described.

The communication terminals 2 to 4 at the sites B to D similarlytransmit the participation request to the communication managementsystem 50, and select the virtual meeting room selected by thecommunication terminal 1. Thereby, the communication terminals 1 to 4participate in the same communication session to have a video call witheach other. The communication session participation method describedabove with FIGS. 23 and 24 is illustrative. For example, thephotographer 8 may establish a communication session by specifyingidentification information of a destination communication terminal oruser. In this case, the communication session is established when thedestination communication terminal or user specified as the destination(i.e., address) responds to a call from the photographer 8.

The attitude information of the image capturing device 5 a will bedescribed with FIGS. 25 to 27C.

FIG. 25 is a diagram illustrating an example of coordinate axes of theimage capturing device 5 a. FIG. 26 is a diagram illustrating an exampleof a reference attitude of the image capturing device 5 a. The referenceattitude of the image capturing device 5 a refers to the attitude of theimage capturing device 5 a in an initial state.

As illustrated in FIG. 25, the longitudinal direction of the imagecapturing device 5 a corresponds to the Z-axis, and the direction ofpassing through the two fisheye lens 102 a and 102 b from the surface ofthe image capturing device 5 a with the shutter button SB to the othersurface of the image capturing device 5 a corresponds to the Y-axis.Further, the width direction of the image capturing device 5 acorresponds to the X-axis. The X-axis, the Y-axis, and the Z-axis movewith the movement of the image capturing device 5 a. The image capturingdevice 5 a is rotatable around each of the X-axis, the Y-axis, and theZ-axis. The rotation angle around the X-axis, the rotation angle aroundthe Y-axis, and the rotation angle around the Z-axis are represented asα, β, and γ, respectively.

As illustrated in FIG. 26, when the photographer 8 points the imagecapturing device 5 a at an object 180, the photographer 8 points theupper side in the longitudinal direction of the image capturing device 5a to the object 180. When the erected state of the image capturingdevice 5 a is defined as the initial state, the rotation angles α, β,and γ in the state of FIG. 26 are −90 degrees, 0 degrees, and 0 degrees,respectively.

FIGS. 27A to 27C are diagrams illustrating values detected by theacceleration and orientation sensor 118 of the image capturing device 5a. As illustrated in FIG. 27A, the image capturing device 5 a rotatesaround the X-axis by the rotation angle α. When the values detected bythe acceleration and orientation sensor 118 are represented as (ax, ay,az), the rotation angle α is expressed as:

$\begin{matrix}{\alpha = {a\;{\tan^{2}\left( {{ay},{- {az}}} \right)}}} & {{{{ay}^{\bigwedge}2} + {{az}^{\bigwedge}2}} \geq {{threshR}^{\bigwedge}2}} \\{= 0} & {{{{ay}^{\bigwedge}2} + {{az}^{\bigwedge}2}} < {{threshR}^{\bigwedge}2}}\end{matrix}$

Herein, the range of values of the rotation angle α is expressed as−π<α≤π, and threshR represents a threshold value set as desired.

The rotation around the Y-axis and the rotation around the Z-axis areillustrated as in FIG. 27B and FIG. 27C, respectively. To improveresponsiveness, the gyro sensor 119 may be used. Values (gα, gβ, gγ)output from the gyro sensor 119 correspond to an angular velocity(radians per second (rad/sec)). With the values (gα, gβ, gγ) of the gyrosensor 119, the rotation angles α, β, and γ (i.e., the attitudeinformation) of the image capturing device 5 a may be calculated as:α(n+1)=α(n)+k*gα*dtβ(n+1)=β(n)+k*gβ*dtγ(n+1)=γ(n)+k*gγ*dt

Herein, the direction of the image capturing device 5 a in the initialstate is expressed as (α(0), β(0), γ(0))=(α0, β, γ). Further, krepresents a coefficient related to the sensitivity of the gyro sensor119, and the initial value of the coefficient k is set to 1.0. To reducethe influence of movements of a hand holding the image capturing device5 a, for example, the initial value of the coefficient k may be adjustedto 0.5, for example, to obtain the effect of a low-pass filter. Thethus-obtained values (α, β, γ) represent the attitude information. Therotation angle γ around the Z-axis does not affect the pointingdirection of the image capturing device 5 a, and thus may not becalculated.

The correct value of the rotation angle α around the X-axis isobtainable from the signal of the acceleration and orientation sensor118 even if the initial value of the rotation angle α is not zero.Further, the rotation angle γ around the Z-axis does not affect thepointing direction of the image capturing device 5 a, as describedabove. As for the rotation angle β around the Y-axis, on the other hand,the initial value thereof is simply set to zero. Therefore, if the imagecapturing device 5 a is tilted in the initial state at the time ofpower-on, for example, the correct value of the rotation angle β may notbe obtained. When the photographer 8 points the image capturing device 5a at an object, therefore, it is desirable that the photographer 8 firstinitializes the values of the rotation angles α, β, and γ by pressingand holding a predetermined button of the image capturing device 5 a inthe erected state of the image capturing device 5 a, for example.

Alternatively, the initial value of the rotation angle β around theY-axis may also be calculated based on the output from the accelerationand orientation sensor 118 similarly to the value of the rotation angleα around the X-axis.

With reference to FIGS. 28A to 28C, a description will be given of howthe coordinates of the point of interest used by the photographer 8 topoint at the object are processed in the zenith correction.

FIG. 28A illustrates a state in which the photographer 8 points theimage capturing device 5 a at an object, specifically a doll 190. Forexample, when the coordinates of the point of interest represent pixelsor an area corresponding to the image of the subject located on theupper side in the longitudinal direction of the image capturing device 5a, the image captured in the state of FIG. 28A is rendered as in FIG.28B. FIG. 28B illustrates an equidistant cylindrical image as anomnidirectional image expressed by equidistant cylindrical projection.This equidistant cylindrical image is not subjected to the zenithcorrection. In the formation of the equidistant cylindrical image, thesubject located on the upper side in the longitudinal direction of theimage capturing device 5 a is extended in the lateral direction in anupper end part of the equidistant cylindrical image. That is, withoutthe zenith correction, the subject located on the upper side in thelongitudinal direction of the image capturing device 5 a is placed inthe upper end part of the equidistant cylindrical image. Further, in theequidistant cylindrical image, the subject in the upper end part thereofis extended in the lateral direction. Without the zenith correction, thecoordinates of the point of interest remain at the same position in theequidistant cylindrical image.

Irrespective of the attitude of the image capturing device 5 a, anobject located in an upper part of an actual space is placed in an upperpart of the image by the image capturing device 5 a. Similarly, anobject located in a lower part of the actual space is placed in a lowerpart of the image by the image capturing device 5 a. Therefore, thezenith correction is performed on the image based on the attitudeinformation. FIG. 28C illustrates an equidistant cylindrical imagesubjected to the zenith correction. The direction and amount of thezenith correction are determined by the attitude information. Therefore,the coordinates of the point of interest in the equidistant cylindricalimage are obtained by rotating the coordinates of the original definedpoint of interest with coordinate conversion information for rotatingthe image for the zenith correction. In FIG. 28C, the coordinates of thepoint of interest are encircled by a circle 191. The circle 191 includesthe image of the doll 190 illustrated in FIG. 28A. That is, the objectpointed by the photographer 8 is identified with the coordinates of thepoint of interest.

Functions of the image processing unit 16 a of the image capturingdevice 5 a will be described with FIG. 29.

FIG. 29 illustrates major functional blocks of the image processing unit16 a of the image capturing device 5 a. The image processing unit 16 aincludes a captured image acquiring unit 202, a combining unit 204, azenith correction unit 206, an omnidirectional image generating unit208, an image compression unit 210, a point-of-interest defining unit194, a point-of-interest converting unit 196, and a point-of-interestspecification determining unit 198.

The captured image acquiring unit 202 controls the two imaging elements103 a and 103 b to acquire therefrom the respective captured images.When the image capturing device 5 a captures still images, two capturedimages are acquired for one frame when the shutter button SB is pressed.When the image capturing device 5 a captures video images, images aresequentially captured in successive frames, and two captured images areacquired for each of the frames. The image captured by each of theimaging elements 103 a and 103 b is a fisheye image with a substantiallyhemispherical field of view, and is a partial image of theomnidirectional image. In the following description, the image capturedby each of the imaging elements 103 a and 103 b will be occasionallyreferred to as the partial image.

The combining unit 204 executes a combining position detection processof detecting a combining position for combining the two acquired partialimages, to thereby combine the two partial images at the combiningposition. In the combining position detection process, the combiningunit 204 detects, for each frame, respective positional deviationamounts of a plurality of corresponding points in an overlapping area ofthe partial images.

The point-of-interest defining unit 194 defines and holds thecoordinates (x, y) of the point of interest in a planar image. Thecoordinates of the point of interest are not set by a user (e.g., thephotographer 8), but are previously defined (i.e., fixed) in amanufacturing, designing, or shipment process, for example. Plural setsof coordinates may be defined for the point of interest. In this case,the photographer 8 operates the image capturing device 5 a or thecommunication terminal 1 to select a desired set of coordinates for thepoint of interest.

The point-of-interest specification determining unit 198 determineswhether the photographer 8 is specifying the point of interest. Forexample, if a predetermined button of the image capturing device 5 a iskept pressed down, the point-of-interest specification determining unit198 determines that the photographer 8 is specifying the point ofinterest. If the predetermined button is not kept pressed down, thepoint-of-interest specification determining unit 198 determines that thephotographer 8 is not specifying the point of interest.

Based on the attitude information acquired by the attitude informationacquiring unit 15 a, the zenith correction unit 206 executes acorrection process for adjusting the zenith direction of theomnidirectional image to match a predetermined reference direction.Specifically, the zenith correction unit 206 corrects a later-describedconversion table in FIG. 30A. Herein, the predetermined referencedirection is typically along the vertical direction in which theacceleration of gravity acts. With the zenith direction of theomnidirectional image corrected to match the vertical direction (i.e.,upward direction), the user is prevented from having discomfort such assickness from watching three-dimensional image, particularlythree-dimensional video image, when the field of view is changed duringviewing of the image.

Based on the processing results of the point-of-interest defining unit194 and the point-of-interest specification determining unit 198 and thecorrected conversion table, the point-of-interest converting unit 196converts the coordinates (x, y) of the point of interest in a planecoordinate system (hereinafter referred to as the plane coordinates (x,y)) into the coordinates (θ, φ) of the point of interest in a sphericalcoordinate system (hereinafter referred to as the spherical coordinates(θ, φ)). That is, the point-of-interest converting unit 196 determinesthe coordinates of the point of interest subjected to the zenithcorrection based on the attitude information. The point-of-interestconverting unit 196 sets the spherical coordinates (θ, φ) of the pointof interest in the point-of-interest information.

TABLE 1 given below illustrates an example of the point-of-interestinformation.

TABLE 1 unit or value example θ of the point of interest pixel 500 φ ofthe point of interest pixel 300 specification or non-specificationspecified or specified of the point of interest unspecified

The point-of-interest information includes the spherical coordinates (θ,φ) of the point of interest and information of whether the point ofinterest is specified or unspecified. If the point of interest isspecified, the point-of-interest specification determining unit 198 setsa value “specified” in the field of specification or non-specificationof the point of interest. If the point of interest is unspecified, thepoint-of-interest specification determining unit 198 sets a value“unspecified” in the field of specification or non-specification of thepoint of interest. The point of interest per se is constantly includedin the point-of-interest information.

The omnidirectional image generating unit 208 executes a process ofgenerating the omnidirectional image from the two captured partialimages with the processing result of the point-of-interest convertingunit 196. In the present embodiment, the conversion table is also usedto generate the omnidirectional image from the two partial images. Theomnidirectional image generating unit 208 generates the omnidirectionalimage from the two partial images with the corrected conversion table.With this process, a processing load for obtaining the finalomnidirectional image is reduced.

The method of generating the omnidirectional image, however, is notlimited to the above-described method. For example, the two partialimages may be combined to generate an omnidirectional image, and thezenith correction process may be performed on the thus-generatedomnidirectional image to generate an omnidirectional image subjected tothe zenith correction.

The image compression unit 210 includes a still image compression unit.When the image capturing device 5 a captures still images, the imagecompression unit 210 compresses the captured image into image data in apredetermined still image format such as the joint photographic expertsgroup (JPEG) format. When the image capturing device 5 a captures videoimages, the image compression unit 210 compresses successive frames ofthe captured image into image data in a predetermined video imageformat. Video compression formats usable in this case include, but arenot limited to, H.264/moving picture experts group (MPEG)-4 advancedvideo coding (AVC), H.265/high efficiency video coding (HEVC), motionJPEG, and motion JPEG 2000, for example. The generated image data istransmitted to the sites B to D by a transmission unit 211. Thetransmission unit 211 corresponds to the communication unit 18 a of theimage capturing device 5 a and the transmitting and receiving unit 91 ofthe communication terminal 1.

Functions of the display control units 94, 74, 34 c, and 34 d of thecommunication terminals 1 to 4 will be described.

In response to receipt of an omnidirectional image from another site,the communication terminals 1 to 4 display the omnidirectional image. Asillustrated in FIG. 29, the display control unit 74 of the communicationterminal 2 includes an image deploying unit 212, an image rotating unit216, a cutout unit 220, a scaling and letterboxing unit 222, and anoutput unit 224. Although the functions of the display control unit 74of the communication terminal 2 will be described below with FIG. 29,the description similarly applies to the display control unit 34 c ofthe communication terminal 3, the display control unit 34 d of thecommunication terminal 4, and the display control unit 94 of thecommunication terminal 1.

The image deploying unit 212 reads and acquires the omnidirectionalimage transmitted from the image capturing device 5 a, and deploys theacquired omnidirectional image on a memory of the communication terminal2.

The image rotating unit 216 rotates the omnidirectional image inaccordance with the point of interest determined by thepoint-of-interest converting unit 196. Thereby, the coordinates of thepoint of interest are moved to the center of the equidistant cylindricalimage. A detailed description of the rotation process will be describedlater with FIG. 32.

The cutout unit 220 cuts out a part (e.g., a central part) of therotated omnidirectional image to generate a cut-out image. Cutting out apart of an image refers to taking out a certain part of the image, andmay also be described as trimming. The cutout unit 220 preferably cutsout a central part of the converted omnidirectional image, to therebycut out an image corresponding to a part of the omnidirectional imagehaving a certain size and centering around the point of interest.

In the present embodiment, the cutout unit 220 has a function ofgenerating a cut-out image by cutting out a part of an image. As well asthis function, the cutout unit 220 may also have a function of reducingthe resolution of the image.

The scaling and letterboxing unit 222 executes an enlargement process onthe image cut out by the cutout unit 220 in accordance with theresolution and aspect ratio of an image output device (e.g., a displayor projector) to which the image is output. The scaling and letterboxingunit 222 further executes a process of adding black bars to the upperand lower sides of the cut-out image to generate an image for display.The output unit 224 outputs (e.g., displays), via an image outputinterface of the communication terminal 2, the image for displaygenerated through the processing of the scaling and letterboxing unit222. The processing of the scaling and letterboxing unit 222 may beomitted if the resolution and aspect ratio of the cut-out image matchthose of the image output device.

In the case of the still image, the image output process by theabove-described functional units (i.e., the image rotating unit 216, thecutout unit 220, the scaling and letterboxing unit 222, and the outputunit 224) is repeated on the same omnidirectional image at least everytime the point of interest changes, or typically at each predeterminedtime interval, to update the image for display in accordance with thepoint of interest at the time of the change in the point of interest orat the each predetermined time interval. In the case of the video image,the image output process by the above-described functional units istypically repeated on the omnidirectional image in each of frames toupdate the image for display.

According to the embodiment, when the photographer 8 tilts or rotatesthe image capturing device 5 a in a certain direction relative to thedirection of the image capturing device 5 a in the erected state, thepoint of interest is changed to enable the user to view theomnidirectional image displayed in accordance with the changed point ofinterest.

The conversion table will be described in more detail with FIGS. 30A and30B.

FIG. 30A is a diagram illustrating an example of the conversion table.FIG. 30B is a diagram illustrating conversion from a plane coordinatesystem into a spherical coordinate system. FIG. 30A illustrates aconversion table used by the image capturing device 5 a of theembodiment. The conversion table specifies projection of the partialimage captured by the imaging element 103 a or 103 b, which is expressedby the plane coordinates (x, y), to the equidistant cylindrical imageexpressed by the spherical coordinates (θ, φ) (hereinafter referred toas the corrected image). For each of the fisheye lenses 102 a and 102 b,the conversion table holds association information for all values of thespherical coordinates (θ, φ). The association information associates thevalues of the spherical coordinates (θ, φ) of the corrected image withthe values of the plane coordinates (x, y) of the pre-correction partialimage, which are mapped into the values of the spherical coordinates (θ,φ). In the example illustrated in FIGS. 30A and 30B, the angle of eachpixel is 0.1 degrees in both the θ direction and the φ direction, andthe conversion table stores information of 3600×1800 correspondingrelationships for each of the fisheye lenses 102 a and 102 b. Theoriginal conversion table may be tabulated with values calculated withdistortion from an ideal lens model previously corrected by amanufacturer of the image capturing device 5 a, for example.

As illustrated in FIG. 30B, coordinates 199A of the point of interestexpressed as the plane coordinates (x, y) are projected to the sphericalcoordinates (θ, φ) based on the conversion table. As viewed from theimage capturing device 5 a, the direction of the coordinates 199A of thepoint of interest expressed as the plane coordinates (x, y) is constant.If the conversion table is fixed, therefore, coordinates 199B of thepoint of interest expressed as the spherical coordinates (θ, φ) are alsoconstant.

However, the attitude of the image capturing device 5 a changesdepending on how the photographer 8 holds the image capturing device 5a. Therefore, the vertical direction of the actual space and thevertical direction of the corrected image do not match unless theconversion table is corrected in accordance with the attitudeinformation. In the zenith correction, therefore, the conversion tableis corrected in accordance with the attitude information.

FIG. 31 is a diagram illustrating correction of the conversion tablebased on the attitude information. Herein, three-dimensional orthogonalcoordinates before the coordinate conversion are expressed as (x1, y1,z1), and spherical coordinates of the three-dimensional orthogonalcoordinates (x1, y1, z1) are expressed as (θ1, φ1). Further,three-dimensional orthogonal coordinates after the coordinate conversionare expressed as (x2, y2, z2), and spherical coordinates of thethree-dimensional orthogonal coordinates (x2, y2, z2) are expressed as(θ2, φ2).

In the process of correcting the conversion table, the sphericalcoordinates (θ1, φ1) are converted into the spherical coordinates (θ2,φ2) with equations (1) to (6) given below.

$\begin{matrix}{\mspace{79mu}\left\lbrack {{Math}.\mspace{14mu} 1} \right\rbrack} & \; \\{\mspace{79mu}{{x\; 1} = {{\sin({\phi 1})}{\cos({\theta 1})}}}} & (1) \\{\mspace{79mu}{{y\; 1} = {{\sin({\phi 1})}{\sin({\theta 1})}}}} & (2) \\{\mspace{79mu}{{z\; 1} = {\cos({\phi 1})}}} & (3) \\{\begin{pmatrix}{x\; 2} \\{y\; 2} \\{z\; 2}\end{pmatrix} = {\begin{pmatrix}1 & 0 & 0 \\0 & {\cos\;\alpha} & {\sin\;\alpha} \\0 & {{- \sin}\;\alpha} & {\cos\;\alpha}\end{pmatrix}\begin{pmatrix}{\cos\;\beta} & 0 & {{- \sin}\;\beta} \\0 & 1 & 0 \\{\sin\;\beta} & 0 & {\cos\;\beta}\end{pmatrix}\begin{pmatrix}{\cos\;\gamma} & {\sin\;\gamma} & 0 \\{{- \sin}\;\gamma} & {\cos\;\gamma} & 0 \\0 & 0 & 1\end{pmatrix}\begin{pmatrix}{x\; 1} \\{y\; 1} \\{z\; 1}\end{pmatrix}}} & (4) \\{\mspace{59mu}{{\phi 2} = {{Arccos}\left( {z\; 2} \right)}}} & (5) \\{\mspace{59mu}{{\theta 2} = {{Arctan}\left( \frac{y\; 2}{x\; 2} \right)}}} & (6)\end{matrix}$

To perform rotational transformation with three-dimensional orthogonalcoordinates, a process of converting the spherical coordinates (θ1, φ1)into the three-dimensional orthogonal coordinates (x1, y1, z1) is firstexecuted with equations (1) to (3).

Then, based on equation (4), the three-dimensional orthogonalcoordinates (x1, y1, z1) are converted into the three-dimensionalorthogonal coordinates (x2, y2, z2) with the rotation angles α, β, and γ(i.e., the attitude information) of the image capturing device 5 a thatpoints at the object. Equation (4) indicates that the originalcoordinates are rotated around the X-axis by the rotation angle α,rotated around the Y-axis by the rotation angle β, and rotated aroundthe Z-axis by the rotation angle γ to obtain the post-conversioncoordinates.

Finally, the post-conversion three-dimensional orthogonal coordinates(x2, y2, z2) are converted back to the spherical coordinates (θ2, φ2)with equations (5) and (6). The coordinates (θ1, φ1) represent thepre-correction spherical coordinates of the conversion table, and thespherical coordinates (θ1, φ1) of the conversion table are corrected to(θ2, φ2) in accordance with the attitude information of the imagecapturing device 5 a.

With the corrected conversion table, the point-of-interest convertingunit 196 converts the plane coordinates of the point of interest intothe spherical coordinates of the point of interest. Thereby, thespherical coordinates of the point of interest subjected to the zenithcorrection are obtained.

The image rotation process of the image rotating unit 216 will bedescribed with FIG. 32.

FIG. 32 is a diagram schematically illustrating the image rotationprocess of the image rotating unit 216. Image rotation refers to aprocess of moving the spherical coordinates of the point of interest tothe center of the image. Herein, new coordinates obtained through theimage rotation are expressed as (θ_(N), φ_(N)), and the coordinates ofthe point of interest are expressed as (θ₀, φ₀). Further, givencoordinates to be rotated are expressed as (θ, φ). With equations (7)and (8) given below, the coordinates of the point of interest are movedto the center of the image.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 2} \right\rbrack & \; \\\begin{matrix}{\theta_{N} = {\theta - \theta_{0} + {180{{^\circ}\left( {{\theta - \theta_{0}} \leq {180{^\circ}}} \right)}}}} \\{= {\theta - \theta_{0} - {180{{^\circ}\left( {{\theta - \theta_{0}} > {180{^\circ}}} \right)}}}}\end{matrix} & (7) \\\begin{matrix}{\varphi_{N} = {\varphi - \varphi_{0} + {90{{^\circ}\left( {{\varphi - \varphi_{0}} \leq {90{^\circ}}} \right)}}}} \\{= {\varphi - \varphi_{0} - {90{{^\circ}\left( {{\varphi - \varphi_{0}} > {90{^\circ}}} \right)}}}}\end{matrix} & (8)\end{matrix}$

With equations (7) and (8), a given pixel is rotated (i.e., moved) bythe difference between the coordinates of a center point and thecoordinates of the point of interest in the θ direction and the φdirection, as indicated by a broken arrow in FIG. 32.

With reference to FIG. 33 a description will be given of a procedure ofa process of transmitting the omnidirectional image and the audio dataobtained at the site A to the communication terminals 2 to 4 via thecommunication management system 50.

FIG. 33 is a sequence diagram illustrating an exemplary procedure of aprocess of communicating the omnidirectional image and the audio data ina video call. The communication unit 18 a of the image capturing device5 a first transmits the omnidirectional image and the audio data to thecommunication unit 98 of the communication terminal 1 (step S101). Theomnidirectional image is obtained from the captured image of a subject(e.g., an object or surroundings). The audio data is obtained fromcollected sounds. In this step, the image capturing device 5 a attachesthe point-of-interest information to the omnidirectional imageirrespective of the result of determination by the point-of-interestspecification determining unit 198. Then, the communication unit 98 ofthe communication terminal 1 receives the omnidirectional image and theaudio data.

Then, the transmitting and receiving unit 91 of the communicationterminal 1 transmits to the communication management system 50 theomnidirectional image, the audio data, and the point-of-interestinformation transmitted from the image capturing device 5 a (step S102).The image data ID for identifying the captured image data of theomnidirectional image to be transmitted is also transmitted in thisstep. Then, the transmitting and receiving unit 51 of the communicationmanagement system 50 receives the omnidirectional image with the imagedata ID, the audio data, and the point-of-interest information.

Then, the transmitting and receiving unit 51 of the communicationmanagement system 50 transmits, to the communication terminals 2 to 4participating in the video call participated in by the communicationterminal 1, the omnidirectional image with the image data ID, the audiodata, and the point-of-interest information (steps S103, S104, andS105). The image data ID transmitted here is for identifying theomnidirectional image to be transmitted. Then, the transmitting andreceiving unit 71 of the communication terminal 2, the transmitting andreceiving unit 31 c of the communication terminal 3, and thetransmitting and receiving unit 31 d of the communication terminal 4receive the omnidirectional image with the image data ID, the audiodata, and the point-of-interest information. Normally, the communicationterminal 1 also receives the omnidirectional image of the site A fromthe communication management system 50 and displays the omnidirectionalimage, although illustration thereof is omitted in FIG. 33.

With FIGS. 34A and 34B, a detailed description will be given of aprocess of generating the omnidirectional image performed by the imagecapturing device 5 a and the process of displaying the omnidirectionalimage performed by the communication terminals 2 to 4.

FIG. 34A is a flowchart illustrating an exemplary process of generatingthe omnidirectional image performed by the image capturing device 5 a.FIG. 34B is a flowchart illustrating an exemplary process of displayingthe omnidirectional image performed by the communication terminals 2 to4. In FIG. 34A, which illustrates a process of the image capturingdevice 5 a, illustration of the communication terminal 1 is omitted.

The point-of-interest defining unit 194 defines the plane coordinates ofthe point of interest (step S200). As described above with FIG. 29, theplane coordinates of the point of interest are statically preset.

During an image capturing process, the point-of-interest specificationdetermining unit 198 determines, based on sensor information, whetherthe photographer 8 is specifying the point of interest (step S201). Thesensor information refers to information indicating whether pressing ofa predetermined button of the image capturing device 5 a has beendetected. When pressing of the predetermined button is detected, thepoint-of-interest specification determining unit 198 determines that thepoint of interest is specified. When the point of interest is specified,the point-of-interest specification determining unit 198 sets the value“specified” in the field of specification or non-specification of thepoint of interest in TABLE 1.

The captured image acquiring unit 202 acquires the captured images fromthe imaging elements 103 a and 103 b (step S202).

Then, the combining unit 204 detects the combining position in theoverlapping area of the two acquired partial images, and reflects aresult of detection of the combining position in the conversion table(step S203). With the result of detection of the combining position, theconversion table illustrated in FIG. 30A is corrected such that thevalues of the plane coordinates (x, y) of the partial images reflectingthe correction of the combining position are associated with the valuesof the spherical coordinates (θ, φ) of the corrected image. The zenithcorrection unit 206 corrects the conversion table based on the attitudeinformation (step S204). That is, the zenith correction unit 206executes the zenith correction.

With the corrected conversion table, the point-of-interest convertingunit 196 then converts the plane coordinates of the point of interestdefined by the point-of-interest defining unit 194 into the sphericalcoordinates of the point of interest (step S205). The point-of-interestconverting unit 196 further sets, in the point-of-interest informationin TABLE 1, the spherical coordinates of the point of interest obtainedthrough the conversion. The coordinates of the point of interest arealso set in the point-of-interest information when it is determined atstep S201 that the point of interest is unspecified. Alternatively, thecoordinates of the point of interest may not be set in thepoint-of-interest information when it is determined at step S201 thatthe point of interest is unspecified.

The omnidirectional image and the point-of-interest informationgenerated through the above-described process are transmitted to thecommunication terminals 2 to 4, as described above with FIG. 33.

FIG. 34B is a flowchart illustrating a process performed by each of thedisplay controlling units 74, 34 c, and 34 d of the communicationterminals 2, 3 and 4. The image rotating unit 216 first determineswhether the point of interest is “specified” in the point-of-interestinformation attached to the omnidirectional image (step S210).

If the point of interest is “specified” in the point-of-interestinformation (Yes at step S210), the image rotating unit 216 rotates theomnidirectional image such that the point of interest is positioned atthe center of the equidistant cylindrical image (step S211). Thereby,the point of interest is forcibly displayed irrespective of thepredetermined area image displayed by the communication terminal 2, 3,or 4 until the receipt of the point-of-interest information indicatingthat the point of interest is specified.

Then, the cutout unit 220 cuts out a central part of the omnidirectionalimage to generate a cut-out image (step S212). The size of the image tobe cut out from the omnidirectional image is previously determined. Inthe example of FIG. 8, the center point CP corresponds to thecoordinates of the point of interest, and the angle of view a and thedistance f are previously determined.

If the point of interest is “unspecified” in the point-of-interestinformation (No at step S210), the omnidirectional image may be rotatedas desired by the user of the communication terminal 2, 3, or 4. If theomnidirectional image is a still image, the predetermined area Tgenerated based on the last user operation of rotating theomnidirectional image is displayed. If the omnidirectional image is avideo image, the predetermined area T generated based on the last useroperation of rotating the omnidirectional image is kept displayed. Thecutout unit 220 cuts out the predetermined area T determined based onthe user operation (step S213).

The scaling and letterboxing unit 222 enlarges the cut-out image inaccordance with the resolution and aspect ratio of the image outputdevice to which the image is output, and adds black bars to the enlargedimage, to thereby generate an image for display (step S214).

The output unit 224 outputs the generated image for display to the imageoutput device via the image output interface (step S215).

An example of display of the image on a communication terminal will bedescribed.

FIG. 35 illustrates an example of an image display screen 250 displayedon the display 917 of the communication terminal 2 at the site B. A leftdisplay area L1 of the image display screen 250 displays theomnidirectional image of the site A. An upper-right display area L2 ofthe image display screen 250 displays the omnidirectional image of thesite C. A middle-right display area L3 of the image display screen 250displays the image of the site D. A lower-right display area L4 of theimage display screen 250 displays the image of the site B, at which thecommunication terminal 2 is located. The display area L1 is a maindisplay area, and the display areas L2 to L4 are sub-display areas. Eachof the communication terminals 1 to 4 is capable of switching the imageof the main display area to one of the images of the sub-display areas.At each of the sites A to D, the main display area normally displays theimage of the site at which a keyperson of the video call is located.

The display areas L1 and L2 display an omnidirectional image icon 192.The omnidirectional image icon 192 indicates that the image displayed inthe corresponding display area is the omnidirectional image. The user 9b of the communication terminal 2 is able to change the predeterminedarea T. The display area L1 further displays a point-of-interest icon193. The point-of-interest icon 193 is displayed by the display controlunit 74 when the point of interest is specified in the point-of-interestinformation. The point-of-interest icon 193 indicates that the point ofinterest is specified. The point-of-interest icon 193 is an example of amessage indicating that the point of interest is specified. Thereby, theuser 9 b understands that the point of interest is currently displayed,and that the predetermined area T is not allowed to be changed.

That the predetermined area T is not allowed to be changed means thatthe receiving unit 72 does not accept a change in the predetermined areaT displayed in the display area, or that the receiving unit 72 accepts achange in the predetermined area T displayed in the display area, butwhen the user 9 b stops the operation of changing the predetermined areaT displayed in the display area, the point of interest converted by thepoint-of-interest converting unit 196 is cut out from theomnidirectional image by the cutout unit 220 and displayed again. Forexample, the user 9 b is able to display a desired predetermined area Tin the display area while dragging the predetermined area T with a mouseof the communication terminal 2 kept clicked on the predetermined area Tor while swiping the predetermined area T with a finger of the user 9 bkept touched on the predetermined area T.

Alternatively, the user 9 b may be allowed to stop the display of thepoint of interest. In this case, the communication terminal 2 isequipped with a predetermined button, for example. When the user 9 bpresses the predetermined button once, the display of the point ofinterest stops, allowing the user 9 b to display a desired predeterminedarea T. Further, when the user 9 b presses the predetermined buttonagain, the point of interest is automatically displayed.

According to the embodiment, video communication between multiple sitesis possible, as illustrated in FIG. 35. Therefore, when the users 9 band 9 d know each other but are unable to visit a real estate propertyfor viewing or visit a real estate agency together, for example, theusers 9 b and 9 d are able to experience realistic viewing. When theusers 9 b and 9 d are a couple and one of the users 9 b and 9 d isunable to visit the real estate agency, for example, the users 9 b and 9d are able to see and comment on the same property from the differentsites B and D.

As described above, according to the image communication system 10 ofthe embodiment, the coordinates of the point of interest are previouslydefined in the image capturing device 5 a. The photographer 8 points theimage capturing device 5 a at the target object such that thecoordinates of the point of interest are aligned with the target object.With the image capturing device 5 a alone, therefore, the photographer 8is able to draw the attention of a user at a different site to the pointof interest in real time.

In the above-described embodiment, a description has been given of anexample in which the image communication system 10 is used in viewing ofreal estate properties. However, the application of the imagecommunication system 10 is not limited to this example. The imagecommunication system 10 is also applicable to other situations in whichan object at a given site is pointed with the image capturing device 5a, such as an exhibition, a show, a factory tour, sightseeing, and aninspection, for example.

In the above-described embodiment, the object is pointed by a personwith the image capturing device 5 a. Alternatively, the object may bepointed by a machine, robot, or animal. For example, the image capturingdevice 5 a may be fixed to the front side in the traveling direction ofa movable machine to constantly display the image of an object presentin front of the machine in the traveling direction, and if necessary,the image may be rotated to check the surroundings of the machine. Inthis case, switching between display and non-display of the coordinatesof the point of interest may be performed by a device or apparatus thatreceives the omnidirectional image.

Further, in the above-described embodiment, the point of interest isdisplayed as a part of the omnidirectional image, for example. However,the omnidirectional image is not necessary required to be a 360-degreesurrounding image. For example, a single hemispherical image may becaptured, or the direction of capturing the 360-degree image may belimited to the horizontal direction. Further, a planar image with morepixels than the number of pixels covered by one display may be used.

The blocks in the exemplary configurations illustrated in drawings suchas FIGS. 15A to 16B and FIG. 29 are divided in accordance with majorfunctions of the image communication system 10 to facilitate theunderstanding of the processing of the image communication system 10.The present invention is not limited by how the processing units aredivided or the names of the processing units. The processing of theimage communication system 10 may be divided into a larger number ofprocessing units depending on processes to be performed. Further, aprocessing unit of the image communication system 10 may be sub-dividedto include more processes.

The image communication system 10 may include a plurality ofcommunication management systems 50. Further, the functions of thecommunication management system 50 may be dividedly allocated to aplurality of servers. Further, the image communication system 10 mayinclude a relay device that relays the image data and the audio data.

The above-described embodiments are illustrative and do not limit thepresent invention. Thus, numerous additional modifications andvariations are possible in light of the above teachings. For example,elements and/or features of different illustrative embodiments may becombined with each other and/or substituted for each other within thescope of the present invention.

Each of the functions of the described embodiments may be implemented byone or more processing circuits or circuitry. Circuitry includes aprogrammed processor, as a processor includes circuitry. A processingcircuit also includes devices such as an application specific integratedcircuit (ASIC), digital signal processor (DSP), field programmable gatearray (FPGA), and conventional circuit components arranged to performthe recited functions. Further, the above-described steps are notlimited to the order disclosed herein.

The invention claimed is:
 1. An image capturing device, comprising: animaging device configured to capture an image; and circuitry configuredto define a point of interest in the image; determine whether the pointof interest in the image is specified by a user; attach, to the image ina case that the circuitry determines that the point of interest isspecified by the user, coordinate information indicating a specificcoordinate of the point of interest and information indicating that thepoint of interest is specified; attach, to the image in a case that thecircuitry determines that the point of interest is not specified by theuser, the coordinate information indicating the specific coordinate ofthe point of interest; and in a case that the circuitry attaches theinformation to the image, convert the point of interest to generate aconverted point of interest in accordance with attitude information ofthe image capturing device; and cut out a viewable area from the image,the viewable area including the converted point of interest.
 2. Theimage capturing device of claim 1, wherein the circuitry is furtherconfigured to execute a zenith correction on the image with the attitudeinformation; and convert the point of interest with a result of thezenith correction.
 3. The image capturing device of claim 2, wherein theimaging device includes an imaging element, and the circuitry is furtherconfigured to execute the zenith correction on a conversion table withthe attitude information, the conversion table being used to convertcoordinate values of the imaging element into coordinate values in aspherical coordinate system; determine the point of interest accordingto the coordinate values of the imaging element; and convert the pointof interest into coordinate values in a spherical coordinate system withthe conversion table subjected to the zenith correction with theattitude information.
 4. The image capturing device of claim 2, whereinthe imaging device includes an imaging element, and the circuitry isfurther configured to execute the zenith correction to match an upperside of the image with an upper side of an actual space irrespective ofan attitude of the image capturing device; and define the point ofinterest according to coordinate values of the imaging elementcorresponding to an image of an object pointed by an upper side in alongitudinal direction of the image capturing device.
 5. An imageprocessing system, comprising: the image capturing device of claim 1;and a communication terminal configured to display, in a display area ofa display, the viewable area cut out from the image and including theconverted point of interest.
 6. An image processing system, comprising:circuitry configured to acquire an image captured by an image capturingdevice; define a point of interest in the image; determine whether thepoint of interest in the image is specified by a user; attach, to theimage in a case that the circuitry determines that the point of interestis specified by the user, coordinate information indicating a specificcoordinate of the point of interest and information indicating that thepoint of interest is specified; attach, to the image in a case that thecircuitry determines that the point of interest is not specified by theuser, the coordinate information indicating the specific coordinate ofthe point of interest; and in a case that the circuitry attaches theinformation to the image, convert the point of interest to generate aconverted point of interest in accordance with attitude information ofthe image capturing device; cut out a viewable area from the image, theviewable area including the converted point of interest; and control adisplay to display, in a display area, the viewable area cut out fromthe image and including the converted point of interest.
 7. The imageprocessing system of claim 6, wherein in a case that the viewable areacut out from the image and including the converted point of interest isbeing displayed, the circuitry is further configured to reject a changein the displayed viewable area displayed in the display area, or acceptthe change in the displayed viewable area during a user operation ofchanging the displayed viewable area.
 8. The image processing system ofclaim 6, wherein in a case that the image is attached with informationindicating that the point of interest is specified, the circuitrycontrols the display to display a message indicating that the point ofinterest is specified.
 9. An image processing method, comprising:acquiring an image captured by an image capturing device; acquiringattitude information of the image capturing device; defining a point ofinterest in the image; determining whether the point of interest in theimage is specified by a user; attaching, to the image in a case that thedetermining indicates that the point of interest is specified by theuser, coordinate information indicating a specific coordinate of thepoint of interest and information indicating that the point of interestis specified; attach, to the image in a case that the determiningindicates that the point of interest is not specified by the user, thecoordinate information indicating the specific coordinate of the pointof interest; and in a case that the information is attached to theimage, converting the point of interest to generate a converted point ofinterest in accordance with the attitude information; cutting out aviewable area from the image, the viewable area including the convertedpoint of interest; and controlling a display to display, in a displayarea, the viewable area cut out from the image and including theconverted point of interest.
 10. The image capturing device of claim 1,wherein the image is an omnidirectional image.
 11. The image capturingdevice of claim 1, wherein the specific coordinate is athree-dimensional coordinate.
 12. The image capturing device of claim 1,wherein the imaging device includes a wide angle lens.
 13. The imagecapturing device of claim 1, wherein in the case that the circuitryattaches the information to the image, the circuitry is furtherconfigured to rotate the image such that the point of interest ispositioned at a center of image.
 14. The image processing systemaccording to claim 6, wherein the image is an omnidirectional image. 15.The image processing system according to claim 6, wherein the specificcoordinate is a three-dimensional coordinate.
 16. The image processingsystem according to claim 6, wherein the imaging device includes a wideangle lens.
 17. The image processing system according to claim 6,wherein in the case that the circuitry attaches the information to theimage, the circuitry is further configured to rotate the image such thatthe point of interest is positioned at a center of image.